Knowstack

Knowstack is a curated learning platform — think of it as your own private daily.dev, seeded not by user submissions but by the best content already curated for you: developer newsletters. Every week, newsletters like This Week in React, JavaScript Weekly, and Golang Weekly hand-pick the best articles on the web. Knowstack scrapes those newsletter archives, follows every linked article, extracts its content, and runs it through an AI enrichment pipeline — summarize, tag, classify difficulty — turning years of scattered weekly issues into one organized, searchable knowledge repository.

Under the hood, Knowstack is two systems joined by a database. An ingestion pipeline built on Crawlee (the Node.js scraping framework — request queues, auto-retries, politeness controls, Cheerio and Playwright crawlers) discovers and fetches articles, and the AI SDK enriches each one with a structured summary, topic tags, and an embedding vector. A reading app then serves it all: browse by topic or source, bookmark articles into a reading list, and — the interesting part — search two ways: keyword search via Typesense ("that article about useTransition") and semantic search via embeddings ("how do I make React feel faster?"), which finds articles that never contain your words but mean what you asked.

The goal of the hackathon project is to build a working slice of this platform: a Crawlee pipeline that scrapes at least two newsletter archives into articles, an AI SDK stage that summarizes, tags, and embeds each one, and a web app where a signed-in user browses, bookmarks, and searches the repository — with both keyword and semantic search working.

What you are building

At a high level, a user of Knowstack should be able to:

1. Browse a living library — a feed of enriched articles (title, source newsletter, AI summary, tags, difficulty, reading time), filterable by topic, source, and date — accumulated automatically from newsletter issues, not entered by hand.
2. Read the gist first — every article carries a faithful 3–5 sentence AI summary and key takeaways, so the user can decide in ten seconds whether the full read is worth it (the link goes to the original — Knowstack is an index, not a mirror).
3. Search like a search engine — instant, typo-tolerant keyword search over titles, summaries, and tags via Typesense, with filters as facets.
4. Search like a colleague — ask in natural language and get semantically similar articles via embedding search, even with zero keyword overlap.
5. Bookmark & track — save articles to a reading list, mark them read, and organize saves into named collections ("Learning Go", "React performance").
6. Watch it grow — an admin view of the pipeline: sources, last crawl, articles discovered / enriched / failed, and a button to trigger a crawl.

Architecture overview

Knowstack is split into an ingestion side (crawl → extract → enrich → index) that runs in the background, and a serving side (web app + search) that users touch. They meet at PostgreSQL — the source of truth — with Typesense holding a derived, rebuildable search index.

The pieces

1. Source definitions & the issue crawler Each newsletter is a source with a small amount of site-specific code: where the archive lives, how to enumerate issues, and how to pull article links out of an issue page. That's the honest reality of scraping — archives differ (JavaScript Weekly lists issues at /issues, This Week in React posts each issue as a page) — so the design move is a per-source adapter exposing discoverIssues() and extractLinks(issue), with everything downstream fully generic. Crawlee's CheerioCrawler fits here: newsletter pages are static HTML, no browser needed.

2. The article crawler Takes the extracted links — which point anywhere on the web — and fetches each page. This is where Crawlee earns its place over hand-rolled fetch loops: a persistent request queue with URL deduplication, automatic retries with backoff, per-domain rate limiting and concurrency control, session/proxy rotation if needed, and the ability to escalate from CheerioCrawler to PlaywrightCrawler for JavaScript-rendered pages. Politeness is a feature: respect robots.txt, keep concurrency low per domain, and identify yourself with a real user agent.

3. The content extractor Raw article HTML is full of navs, ads, and cookie banners. @mozilla/readability (the engine behind Firefox's Reader View, run over linkedom/jsdom) strips it to title, byline, and clean article text. Compute a reading-time estimate, and guard the pipeline: paywalled pages, videos, and GitHub repos extract poorly — detect thin content and mark those articles link-only instead of feeding junk to the LLM.

4. AI enrichment (the AI SDK stage) Each cleanly extracted article makes one generateObject call — the AI SDK's structured-output mode with a Zod schema — so the model returns typed data, not prose to parse:

const { object } = await generateObject({
  model: 'anthropic/claude-haiku-4-5', // via AI Gateway — cheap + fast fits this job
  schema: z.object({
    summary: z.string().describe('3–5 faithful sentences, no hype'),
    takeaways: z.array(z.string()).max(5),
    tags: z
      .array(z.string())
      .max(6)
      .describe('lowercase topic slugs, e.g. "react", "concurrency"'),
    difficulty: z.enum(['beginner', 'intermediate', 'advanced']),
    contentType: z.enum([
      'tutorial',
      'deep-dive',
      'news',
      'opinion',
      'release-notes',
    ]),
  }),
  prompt: `Summarize and classify this article:\n\n${title}\n\n${cleanText.slice(0, 12000)}`,
})

A second, far cheaper call — embed with an embedding model — turns title + summary into a vector for semantic search. Two rules keep this stage sane: enrich once, store forever (never re-call the LLM for content you already processed), and constrain tags (either close the tag vocabulary in the prompt, or normalize/merge tags in a post-step — otherwise you get react, reactjs, and react.js as three topics).

5. PostgreSQL (the source of truth) The relational backbone: sources, issues, articles (URL, canonical URL, source/issue provenance, extracted text, enrichment fields, embedding, pipeline status), tags + article_tags, and the user side — users, bookmarks (with read/unread state), collections. Two modeling points matter: canonical URLs as the dedup key (the same viral article appears in multiple newsletters — store one article with many issue provenances, after stripping tracking params and resolving redirects), and a status column (discovered → fetched → extracted → enriched → indexed | failed) that makes the pipeline resumable and the dashboard trivial.

6. Typesense (the search engine) A derived index over enriched articles: title, summary, tags, source, difficulty as typo-tolerant, faceted keyword search — and, because Typesense supports vector fields, the same collection holds each article's embedding. That enables three query modes from one engine: pure keyword, pure vector (embed the user's query at request time, k-nearest search), and hybrid — Typesense fuses keyword and vector rankings for you, which is the mode that should power the main search box. (An alternative worth knowing: pgvector keeps vectors in Postgres — fewer moving parts, but you give up fused hybrid ranking and instant facets.) The index is rebuildable from Postgres at any time; treat it as a cache, never as storage.

7. The web app A Next.js app: the feed with facet filters, search-as-you-type hitting Typesense, article cards that link out to the original, bookmarks and collections behind auth, and the pipeline dashboard. Server components read Postgres directly; the search box talks to Typesense with a scoped search-only API key.

8. The scheduler / job queue Crawls are long-running and must not block anything user-facing: a BullMQ queue (Redis) with a worker process runs crawl-source, enrich-article, and index-article jobs — giving retries, concurrency limits, and a place for the dashboard's "run now" button to point. A cron trigger enqueues each source weekly (newsletters are weekly — crawling hourly is rude and pointless).

How an article flows

Understanding this end-to-end path is the heart of the project:

1. Discover — the Monday cron enqueues crawl-source: javascriptweekly. The issue crawler loads the archive, finds issue #713 is new, parses it, and extracts 24 article links.
2. Dedup — each link is canonicalized (tracking params stripped, redirects resolved). 21 are new; 3 already exist from other newsletters — they just gain a new issue provenance row.
3. Fetch — Crawlee's article crawler works through the 21 URLs across 15 different domains: rate-limited per domain, retried on failure. One paywalled page yields thin content and is marked link-only.
4. Extract — readability reduces each page to clean text; reading time is computed; status moves to extracted.
5. Enrich — the worker runs generateObject per article: summary, takeaways, tags (normalized against the existing vocabulary), difficulty, content type — then embed produces the vector. Everything lands in Postgres; status: enriched.
6. Index — each article is upserted into Typesense with its text fields, facets, and embedding; status: indexed.
7. Search — a user types "make react apps feel faster during navigation". Keyword search alone would miss the best match ("Understanding useTransition and concurrent rendering") — hybrid search surfaces it near the top because the embeddings agree even where the words don't.
8. Save — the user bookmarks it into their "React performance" collection, reads it on the original site via the link-out, and marks it read. Next Monday, the library grows again on its own.

Technologies to use

The whole stack is TypeScript / JavaScript end-to-end — pick what your team is fastest in within that ecosystem. The concepts matter more than the exact library.

Concepts to understand

Before writing code, make sure the team is comfortable with these ideas.

Web scraping done right

• Crawlee's model — crawlers, the persistent request queue, handlers per label (ISSUE vs. ARTICLE), automatic retries, and enqueueLinks. Read the Crawlee introduction before writing a crawler.
• Static vs. rendered — Cheerio (fast, plain HTTP) covers most of the web; Playwright (real browser) is the expensive fallback. Choosing per-target matters at hundreds of pages.
• Politeness & legality — robots.txt, per-domain rate limits, honest user agents, and storing summaries + links rather than republishing full content. You're building an index with attribution, not a content mirror.
• Scrapers rot — site markup changes break selectors. Per-source adapters isolate the blast radius; the status column and dashboard make breakage visible instead of silent.

Content extraction

• Boilerplate removal — why readability-style extraction beats hand-written selectors for arbitrary sites, and where it fails (paywalls, SPAs, videos, repos).
• Quality gating — detecting thin/failed extractions (length heuristics) and degrading gracefully to link-only entries instead of enriching garbage.
• Canonicalization — tracking-param stripping, redirect resolution, rel=canonical — the difference between one article and five duplicates.

LLM enrichment with the AI SDK

• Structured output — generateObject + Zod: the schema is the contract, .describe() steers the model, and you never parse free text. This is the single most useful AI SDK pattern to learn.
• Prompting for faithfulness — summaries must compress the article, not advertise it; instruct against hype and hallucinated claims, and truncate input to what the task needs.
• Cost & idempotency — enrichment cost scales with articles × tokens; small fast models are the right tool, results are cached forever in Postgres, and re-runs must skip already-enriched rows.
• Tag vocabulary control — open-ended generation produces near-duplicate tags; constrain via prompt (offer the existing vocabulary) and normalize in code.

Search: keyword, semantic, hybrid

• Why two kinds — keyword search matches terms (great for names and APIs: useTransition); embedding search matches meaning (great for questions). Neither subsumes the other.
• Embeddings — text → vector where distance ≈ semantic similarity; embed once per article, embed the query at search time, nearest-neighbor to rank.
• What to embed — title + summary usually beats full text (focused, cheap, within model limits); consistency between indexed and query embeddings matters more than model choice.
• Hybrid ranking — how Typesense fuses keyword and vector scores (rank fusion), and why hybrid is the right default for the main search box.
• Index as derived data — Postgres owns truth; Typesense is rebuildable. A reindex-all script is twenty lines and saves the demo.

Pipelines as systems

• Idempotency everywhere — every stage keyed on canonical URL + status, safe to re-run after any failure.
• Stage isolation via a queue — crawl, enrich, and index as separate retryable jobs, so one flaky article page can't kill a whole source's run.
• Observability — per-article status, per-source last-run stats, and failure reasons surfaced in the dashboard — a pipeline you can't see is a pipeline you can't demo.

Suggested milestones

A realistic path for a hackathon, from smallest useful slice to full loop:

1. Crawl one source — a Crawlee script for one newsletter: archive → issues → deduplicated, canonicalized article URLs printed to the console. Get the request-queue + adapter shape right here.
2. Fetch & extract — the article crawler + readability into Postgres (Drizzle schema with the status column); the dashboard's raw material now exists.
3. Enrich — the AI SDK stage: generateObject summaries/tags/difficulty and embed vectors stored on each article, idempotently. (Browsing this table in any DB client is already impressive.)
4. The reading app — Next.js feed with tag/source filters, article cards with summaries and link-outs, auth, and bookmarks with read state.
5. Keyword search — Typesense indexing on enrichment + a search-as-you-type box with facets (tags, source, difficulty).
6. Semantic search & close the loop — add the embedding field to Typesense, embed queries at search time, switch the main search to hybrid; add a second newsletter source (the test of your adapter design), collections, and the pipeline dashboard with a "crawl now" button.

Stretch goals

• RSS ingestion — many newsletters ship RSS; a feed-based adapter is cheaper and more robust than scraping where available.
• Reading paths — AI-generated ordered sequences ("Learn Go concurrency: these 8 articles, beginner → advanced") built from tags, difficulty, and embeddings.
• Chat with the library — RAG over the repository: embed the question, retrieve top articles, answer with citations using the AI SDK.
• Personalized feed — rank the feed against the centroid of a user's bookmark embeddings: "more like what you save."
• Weekly digest email — the loop closed with irony: Knowstack sends its own newsletter of the week's best additions per user's topics.
• Related articles — nearest-neighbor lookups on each article page.
• Browser extension — "Save to Knowstack" from any page, running the same extract-and-enrich pipeline on demand.
• Trend surfacing — what topics are heating up across sources over time, from tag frequencies per week.