Reader: Add Web Article Extraction to a Node.js Application

Adding web articles to search or summarization becomes awkward when every function invents its own loading and extraction behavior. One returns empty text, another throws, and browser ownership becomes unclear.

This tutorial uses vakra-dev/reader, specifically the local Node package rather than a similarly named hosted API. It converts explicit URLs into uniform records containing title, Markdown, provenance, collection time, and status.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• DOC_URL_1: the full address of the Python tutorial introduction.html page.
• DOC_URL_2: the full address of controlflow.html in the same directory.

Set these in the same Bash/WSL terminal used for subsequent commands:

export DOC_URL_1='<paste-the-actual-doc-url-1>'
export DOC_URL_2='<paste-the-actual-doc-url-2>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install the local dependencies

Use Node.js 24 in a new directory:

mkdir article-reader
cd article-reader
npm init -y
npm install @vakra-dev/reader@0.3.5
npx playwright install chromium

Reader runs its collection logic locally, but requires an installed browser. A successful package installation does not establish that Chromium is available. Resolve browser installation before diagnosing empty article output.

Keep the lockfile. Existing Playwright dependencies, browser caches, and runtime accounts can affect where the browser is found.

Define the record contract

The example accepts a document only when the expected HTTP status, title, and nonempty Markdown are present. Every failed input retains its requested URL.

Requested URL and Reader's base URL are separate fields. A canonical URL is not automatically a safe replacement: it may identify another version. Collection time also differs from the document's publication or modification time.

Run a complete batch adapter

Create read_articles.mts:

import { writeFile } from 'node:fs/promises';
import { ReaderClient } from '@vakra-dev/reader';
 
class ArticleRejected extends Error {
    code: string;
    constructor(code: string) {
        super(code);
        this.name = 'ArticleRejected';
        this.code = code;
    }
}
const urls = process.argv.slice(2);
if (!urls.length) throw new Error('请在命令行提供至少一个文章 URL');
for (const value of urls) {
    if (!['http:', 'https:'].includes(new URL(value).protocol)) {
        throw new Error('只接受 HTTP(S) 页面');
    }
}
const reader = new ReaderClient();
const records: unknown[] = [];
 
try {
    for (const requestedUrl of urls) {
        try {
            const result = await reader.scrape({
                urls: [requestedUrl],
                formats: ['markdown'],
            });
            const item = result.data[0];
            if (!item) throw new ArticleRejected('reader_returned_no_document');
            const title = item.metadata.website.title?.trim();
            const markdown = item.markdown?.trim();
            if (item.metadata.statusCode !== 200) {
                throw new ArticleRejected('unexpected_http_status');
            }
            if (!title || !markdown) throw new ArticleRejected('missing_title_or_body');
            records.push({
                requested_url: requestedUrl,
                source_url: item.metadata.baseUrl,
                title,
                markdown,
                observed_at: item.metadata.scrapedAt,
                engine: item.metadata.engine,
                status: 'accepted',
            });
        } catch (error) {
            records.push({
                requested_url: requestedUrl,
                status: 'failed',
                reason: error instanceof ArticleRejected ? error.code : 'reader_call_failed',
                error_type: error instanceof Error ? error.name : 'UnknownError',
            });
        }
    }
} finally {
    await reader.close();
}
await writeFile('articles.json', JSON.stringify(records, null, 2));
console.log({ inputs: urls.length, records: records.length });

Provide the sources explicitly:

node read_articles.mts   "$DOC_URL_1" "$DOC_URL_2"

These are documentation-page examples, not a guarantee of permanent accessibility. Replace them with your own article URLs. Every input produces either an accepted or failed record.

The loop is sequential to make diagnosis straightforward while reusing one client. Reader also supports multiple URLs per call and batch concurrency. Establish data quality before increasing concurrency according to available resources and target behavior.

Check whether the body is usable

An accepted record has this illustrative structure:

{
  "requested_url": "https://example.16yun.cn/articles/storage",
  "source_url": "https://example.16yun.cn/articles/storage",
  "title": "存储使用指南",
  "markdown": "# 存储使用指南\n\n页面正文……",
  "observed_at": "2026-10-28T09:00:00Z",
  "engine": "由 Reader 返回",
  "status": "accepted"
}

Compare input and output counts, then inspect representative titles and bodies. Check that major sections exist and navigation or login messages have not replaced the article.

Nonempty text is only a minimum check. A long error page can pass it. A fixed-site integration should add source-specific identity evidence such as its article heading, document version, or expected section.

The recorded collection start time comes from Reader. It is not the source's last-modified time. Store those separately when freshness matters.

Give the client a clear owner

A batch creates one client and closes it in finally. A long-running service can instead own a shared client across jobs, closing it when the service stops.

Do not create a browser-backed client for every individual URL unnecessarily. Equally, do not let one request close a shared client while another still uses it.

This command-line example does not expose arbitrary URL fetching to untrusted callers. A later public API needs source-range controls at its boundary; checking only the URL scheme is insufficient to control internal network access.

Add proxies only for an actual network requirement. Empty titles and conversion failures should not automatically be treated as proxy failures.

Troubleshoot

Browser not found: check installation under the same account that runs the program. Host and container browser caches are different environments.

No first data record: inspect Reader's batch errors and controlled logs to distinguish download, browser, and conversion failure. The reason field distinguishes no document, unexpected status, and missing title or body. Library-call failures retain reader_call_failed plus error_type. Redact credentials and sensitive URLs before recording more detail.

Navigation without the article: verify login conditions, dynamic content availability, and extraction scope. Correct material acquisition before increasing concurrency.

HTTP 200 containing a login page: add target identity checks and repair the session. A nonempty title cannot authenticate article identity.

Resources remain after completion: ensure every exit path reaches client cleanup, and check whether other sessions remain owned by the application. Do not terminate all Chrome processes indiscriminately.

Duplicate or old documents appear: inspect language, version, and parameter variants. Preserve provenance and manage document identity explicitly rather than replacing all records with the same title.

A stable adapter tells downstream code which input was processed, what content was obtained, and which failures require attention. That contract remains useful when extraction libraries or target sites change.