Mongoscope

Mongoscope is a terminal-first MongoDB performance analysis tool — think of it as your own hatchet reimagined as an interactive TUI. You point Mongoscope at a mongod log file, and it parses every line, detects slow queries, surfaces missing-index red flags like collection scans, highlights lock contention, and lets you explore all of it from a keyboard-driven terminal interface — no browser, no dashboard server, just your terminal.

Under the hood, Mongoscope leans on the fact that since MongoDB 4.4 all server logs are structured JSON — one JSON document per line, with a stable schema (t timestamp, s severity, c component, attr attributes). That turns "log analysis" from fragile regex archaeology into honest data engineering: parse JSON lines, normalize query shapes, load them into a local SQLite database, and run analytical queries over it. The TUI — built with OpenTUI or Ink, both of which let you build terminal interfaces with React — is the friendly lens on top of that database.

The goal of the hackathon project is to build a working slice of this tool: a CLI that ingests a MongoDB JSON log file into SQLite, and an interactive TUI that ranks slow queries by pattern, flags collection scans and index problems, and lets you drill into any log entry — all without leaving the terminal.

What you are building

At a high level, a user of Mongoscope should be able to:

1. Ingest a log — run mongoscope mongod.log (plain, .gz, or piped from stdin) and watch it parse thousands of lines per second into a local SQLite file.
2. See the big picture — an overview dashboard: time range covered, total operations, slow-query count, top namespaces, error/warning counts, connection churn.
3. Hunt slow queries — a ranked table of query patterns (normalized query shapes, not raw queries) sorted by total/average/max duration, with counts, docs examined vs. returned, and plan summary.
4. Spot index problems — every COLLSCAN, every query where docsExamined dwarfs nreturned, and in-memory sorts, each pointing at a concrete "this namespace needs an index on these fields" suspicion.
5. Drill down — select any pattern to see its individual executions over time, then open the full raw JSON of a single log entry in a detail pane.
6. Filter & search — narrow everything by time window, namespace, severity, component, or free-text search, live from the TUI.

Architecture overview

Mongoscope is split into an ingest pipeline (parse → normalize → store) and an explore layer (analyzers + TUI). The two meet at a SQLite file, which means ingesting is a one-time cost and exploring is instant — reopening a previously ingested log skips straight to the TUI.

The pieces

1. Streaming log reader Log files can be gigabytes. The reader streams line-by-line (node:readline over a file stream, with a gunzip transform for .gz) instead of loading the file into memory, reports progress as it goes, and tolerates the occasional non-JSON line (startup banners, truncated last line) without dying.

2. Log parser Each line is one JSON document in MongoDB's structured log format. The parser turns it into a typed entry: timestamp (t.$date), severity (s: F/E/W/I/D), component (c: COMMAND, WRITE, NETWORK, REPL, CONTROL, …), context/thread (ctx), message (msg), and the grab-bag attr object. The most valuable lines are the ones with msg: "Slow query" — their attr carries durationMillis, ns, planSummary, keysExamined, docsExamined, nreturned, queryHash, lock/storage stats, and the command document itself.

3. Query normalizer (the heart of the tool) Ten thousand slow queries are useless as a list; three slow patterns are actionable. The normalizer takes a command document and replaces every literal value with a placeholder — { user_id: 8231 } and { user_id: 977 } both become { user_id: ? } — so executions of the same shape aggregate into one pattern with count, min/avg/max/total duration. MongoDB's own queryHash helps, but doing your own normalization also covers ops that don't carry one. This is exactly what hatchet does, and it's what turns a log dump into a diagnosis.

4. SQLite store The ingested log lives in a local SQLite file (mongod.log → mongod.db): an entries table for every line, an ops table for slow operations with their metrics broken out into columns, and a patterns table for aggregates. With a few indexes, every TUI view is a simple indexed SQL query — sorting ten thousand patterns by average duration is instant, and re-running Mongoscope on an already-ingested log skips parsing entirely.

5. Analyzers Focused modules that read from SQLite and produce the numbers each view needs:

• Slow queries — patterns ranked by total/avg/max duration, with per-namespace rollups.
• Index issues — planSummary: COLLSCAN occurrences, high docsExamined / nreturned ratios, and hasSortStage in-memory sorts, grouped by namespace and filter shape.
• Locks & contention — operations with long timeAcquiringMicros, write conflicts, and which patterns collide on the same namespace at the same time.
• Connections & errors — connection open/close churn per client IP, and clusters of E/W severity entries.

6. The TUI (OpenTUI or Ink) The entire product surface. Both frameworks let you describe the terminal UI as React components: Ink is the mature option (it renders Jest's and Prettier's output) with flexbox layout via Yoga and a rich ecosystem (ink-table, ink-text-input, ink-select-input); OpenTUI is the newer, rendering-focused option from the SST team. Either way you're building: a tab bar of views, sortable/scrollable tables, a filter input, a JSON detail pane, and a status bar with key hints — all driven by keyboard (j/k to move, Enter to drill in, / to search, s to change sort, q to quit).

How a log line flows

Understanding this end-to-end path is the heart of the project:

1. Read — the streaming reader pulls the next line from the file (or stdin) and hands it to the parser.
2. Parse — the line is JSON-parsed into a typed entry; unparseable lines are counted and skipped. Every entry is inserted into entries.
3. Classify — if the entry is a slow operation (msg: "Slow query"), its metrics (durationMillis, planSummary, keysExamined, docsExamined, nreturned, lock waits) are extracted into an ops row.
4. Normalize — the op's command document has its literals stripped to produce a pattern key; the matching patterns row is upserted with updated count and duration aggregates. Inserts are batched in transactions — this is the difference between ingesting at hundreds vs. tens of thousands of lines per second.
5. Analyze — when ingestion finishes (or on demand), analyzers run their SQL: rank patterns, find COLLSCANs, compute examined-to-returned ratios, bucket errors.
6. Render — the TUI opens on the overview dashboard. The user tabs to Slow Queries, sorts by average duration, and the top row shows find users { email: ? } — 4,200 executions, avg 380 ms, COLLSCAN, 1.2 M docs examined for 1 returned.
7. Drill — Enter shows that pattern's individual executions over time; Enter again opens one execution's full raw JSON in the detail pane. The user now knows exactly which index to create.

Technologies to use

The whole stack is TypeScript / JavaScript end-to-end — pick what your team is fastest in within that ecosystem. The concepts matter more than the exact library.

Concepts to understand

Before writing code, make sure the team is comfortable with these ideas.

MongoDB's structured log format

• JSON lines since 4.4 — every log line is one JSON document; know the envelope fields: t (timestamp), s (severity), c (component), id, ctx, msg, attr.
• The "Slow query" entry — the single most important line type: which attr fields matter (durationMillis, ns, planSummary, keysExamined, docsExamined, nreturned, queryHash, locks, storage) and what each one tells you.
• What "slow" means — slowms (default 100 ms) and the profiler level control what gets logged; your analysis only sees what the server chose to log.
• Components — COMMAND and WRITE carry operations; NETWORK carries connections; REPL, ELECTION, CONTROL tell the story around incidents.

Reading query performance like a DBA

• planSummary literacy — IXSCAN { email: 1 } (used an index) vs. COLLSCAN (read the whole collection) vs. IDHACK (point lookup by _id).
• The examined-to-returned ratio — docsExamined: 1000000, nreturned: 1 is the signature of a missing index; near 1:1 is healthy.
• In-memory sorts — hasSortStage without a supporting index means the server sorted in RAM (and fails past 100 MB).
• Lock metrics — timeAcquiringMicros and write conflicts: the query wasn't slow, it was waiting.
• From symptom to fix — turning a flagged pattern into a concrete recommendation: "create an index on users { email: 1 }".

Query normalization

• Query shape vs. query instance — why aggregation must group by shape, and how literal stripping produces one ({ email: ? }).
• Recursive normalization — handling nested documents, $in arrays (collapse to one placeholder), operators ($gte, $lt keep their key, lose their value), and aggregation pipelines.
• queryHash / planCacheKey — what the server already gives you and where your own normalization must fill the gaps.

Streaming & data engineering

• Streams over buffers — a 5 GB log must be processed line-by-line in constant memory.
• Batched writes — SQLite transactions around every N inserts; the single biggest ingest performance lever.
• Schema for analytics — breaking metrics into typed columns (not JSON blobs) so sorting and filtering are indexed SQL, not full scans.
• Malformed input — counting and skipping bad lines instead of crashing on them.

Building TUIs

• React in the terminal — how Ink/OpenTUI map components to a character grid; flexbox layout without CSS.
• The render loop & performance — never render ten thousand rows; render the visible window (virtualized scrolling) and re-render only on state change.
• Keyboard-driven UX — focus management between panes, vim-style navigation, a discoverable status bar of key hints.
• Terminal constraints — resize handling, color depth, alternate screen buffer, and graceful cleanup on exit (a crashed TUI that corrupts the terminal is a bad demo).

Suggested milestones

A realistic path for a hackathon, from smallest useful slice to full loop:

1. Parse & count — a CLI that streams a real mongod log, parses each JSON line, and prints summary counts (lines, time range, entries by severity and component, slow-query count).
2. Persist to SQLite — ingest entries and slow ops into a schema with proper columns and indexes; batched transactional inserts; re-opening an ingested log skips parsing.
3. Normalize patterns — literal-stripping over command documents; a plain-text "top 10 slowest patterns" report with count, avg/max duration, and plan summary. (At this point you have a useful tool with no TUI at all.)
4. First TUI view — an Ink/OpenTUI app with a scrollable, sortable slow-query table over the SQLite data, plus a status bar with key hints.
5. Drill-down & detail — pattern → executions → raw JSON detail pane; add the index-issues view (COLLSCANs, examined/returned ratios).
6. Polish — overview dashboard with terminal charts (ops over time, duration histogram), filter bar (time / namespace / severity / text), locks and connections views.

Stretch goals

• Live tail mode — mongoscope tail mongod.log follows a growing file and updates the TUI in real time.
• Index advisor — turn COLLSCAN patterns into concrete db.collection.createIndex(...) suggestions, deduplicated against each other.
• Log diffing — ingest two logs (before/after an index change) and compare pattern stats side by side.
• Replica-set awareness — merge logs from multiple nodes; visualize elections, replication lag warnings, and failovers on a timeline.
• Export — dump any view to JSON/CSV/Markdown for sharing in an incident channel.
• Legacy log support — a regex-based parser for pre-4.4 plain-text logs.
• mongosync / Atlas formats — hatchet handles more than mongod logs; supporting a second log source proves the pipeline is generic.