cra-assistent

Document intake

How a published CRA page becomes something this platform can cite. The whole process is below before anything is done to a file — most of it matters, and the step that matters most is the one that can refuse your document.

Public CRA sources only. A document is stored with the canada.ca URL it came from, and one without a URL cannot be stored at all — the column is NOT NULL, so the boundary is in the schema rather than in a reviewer's diligence. No internal CRA material and no taxpayer data. Nothing here fetches anything: files arrive because you saved them.
  1. 1

    Save a CRA page into corpus/raw

    By hand — open the guide or page on canada.ca and save it as HTML or PDF. There is no crawler and there will not be one (DEC-003): canada.ca is behind bot protection, and fetching by hand is the better provenance story. The reading list says which publications the curriculum is asking for.

    See what to fetch
  2. 2

    The scan notices it

    Files are matched on content hash, never on filename. Re-dropping the same document under a new name is a no-op; a file whose bytes changed reads as new even under the old name — which is exactly when CRA has revised the page and it needs looking at again.

  3. 3

    The text and structure come out

    HTML through selectolax, PDF through pypdf, both to the same block stream of headings and paragraphs. The title, CRA's “date modified” footer and the heading outline are pulled out here. PDF heading structure is guessed rather than read, and the report says so.

  4. 4

    It is split into chunks

    A chunk never spans a heading, even when that leaves it short, and chunks never overlap. Both rules exist for what comes later: the heading path is the strongest topic signal in search, and overlapping chunks would make one sentence appear as two agreeing citations.

  5. 5

    Claude reports what it thinks the document is Not built yet

    Which publication, which tax year, whether it looks superseded, and how clean the extraction was. It is a verdict, not a decision — the parser's findings go to the model as observations, and the model's answer comes back to you as one. It is never asked for the source URL.

    Planned — Phase 3, item 6.

  6. 6

    You approve it, with its public source URL Not built yet

    This is the gate. A document with no canada.ca URL cannot be stored, because documents.source_url is NOT NULL — so it can never be embedded, and never cited in an answer. Rejecting and marking superseded are the other two ways out.

    Planned — Phase 3, items 7–8 and 10.

  7. 7

    The approved chunks are embedded Not built yet

    Only then. Nothing reaches the chunks table before a human has approved the document it came from, which is what keeps the corpus and the source boundary the same thing.

    Planned — Phase 3, item 9.

What is in the corpus directory

/app/corpus/raw
0 files on disk 0 not yet imported

Nothing in /app/corpus/raw yet. Save a CRA guide or page into it as HTML or PDF and it will appear here — no restart needed.