Moli: Fetch Pages and Export Content from the Command Line

A short list of pages does not always justify a crawler project. Moli provides command-line page material that can feed document processing or another extraction stage.

This tutorial moves from one Markdown fetch to a batch with an input manifest, content files, and explicit failures. It focuses on output meaning and diagnosis rather than a complete option catalog.

Prepare the input addresses

Locate the following pages in your browser, confirm their contents, and copy their full addresses. Obtain service and repository addresses from account integration settings, official setup instructions, or the repository clone menu. Do not derive real addresses from illustrative output.

• MOLI_INSTALLER_URL: the download address of moli-installer.sh from the official lexmount/moli release.
• DOC_URL_1: the Python tutorial introduction.html address.
• DOC_URL_2: the controlflow.html address in the same directory.

Set these in the same Bash/WSL terminal used for subsequent commands:

export MOLI_INSTALLER_URL='<paste-the-actual-moli-installer-url>'
export DOC_URL_1='<paste-the-actual-doc-url-1>'
export DOC_URL_2='<paste-the-actual-doc-url-2>'

Replace each <VARIABLE_NAME> in JSON or task parameters with the copied value before submission; JSON does not expand environment variables. Any example.16yun.cn source in an output illustration shows field structure, not a demo service deployed for this article.

Install and fetch one page

Use Bash on Linux/macOS, or WSL on Windows. Follow the official installation instructions:

curl --proto '=https' --tlsv1.2 -fsSL   "$MOLI_INSTALLER_URL" | sh
moli --version

The installer selects a released binary. Record the reported version, particularly before upgrading. If the executable is unavailable afterward, inspect the installer's PATH guidance.

Fetch an explicit documentation page:

moli fetch --dump markdown --wait-until done   "$DOC_URL_1" > introduction.md

Inspect the title and main paragraphs. A successful exit and nonempty file are initial checks, not proof that an article rather than a login or error page was retrieved.

Choose an appropriate projection

Markdown suits reading and text processing. A semantic tree is useful when inspecting structure and interaction objects:

moli fetch --dump semantic_tree_text --wait-selector body   "$DOC_URL_1" > introduction-tree.txt

Waiting for body only establishes that body exists. It does not establish that dynamic business data is ready. Adapt the readiness marker to the actual fields.

Structural reading and layout computation are different paths in Moli. DOM or Markdown access does not mean JavaScript is absent. Screenshots and coordinates introduce layout requirements such as --layout; this text task does not need them.

Preserve an outcome for every input

Create urls.txt:

printf '%s\n' "$DOC_URL_1" "$DOC_URL_2" > urls.txt

Create export_pages.sh:

#!/usr/bin/env bash
set -u
batch_dir=$(mktemp -d "./moli-output.XXXXXX")
printf 'id\turl\tfile\n' > "$batch_dir/index.tsv"
printf 'id\turl\texit_code\n' > "$batch_dir/failed.tsv"
index=0
while IFS= read -r url || [ -n "$url" ]; do
  [ -z "$url" ] && continue
  index=$((index + 1))
  name=$(printf 'page-%04d.md' "$index")
  if moli fetch --dump markdown --wait-until done "$url"       > "$batch_dir/$name.part" 2> "$batch_dir/$name.log"; then
    if [ -s "$batch_dir/$name.part" ]; then
      mv "$batch_dir/$name.part" "$batch_dir/$name"
      printf '%s\t%s\t%s\n' "$index" "$url" "$name" >> "$batch_dir/index.tsv"
    else
      printf '%s\t%s\tempty_output\n' "$index" "$url" >> "$batch_dir/failed.tsv"
    fi
  else
    code=$?
    printf '%s\t%s\t%s\n' "$index" "$url" "$code" >> "$batch_dir/failed.tsv"
  fi
done < urls.txt
printf '结果目录：%s\n' "$batch_dir"

Run:

bash export_pages.sh

A fresh directory prevents previous files from masquerading as new results. Each successful, nonempty command output becomes a Markdown file; other outcomes appear in the failure manifest.

Ignoring header rows, accepted plus failed entries should equal nonempty input lines. This script does not discover additional URLs or download attachments.

The input is a locally maintained list of ordinary single-line URLs without tabs. A public service accepting arbitrary addresses needs separate boundary validation.

Interpret the files

The index maps requested addresses to files. Per-page logs hold command diagnostics; partial files retain unaccepted material. Numeric file names are positions within this batch, not long-term document identities.

A redirect can change the actual document while the manifest retains the requested URL. Long-term indexes need explicit redirect, version, and observation metadata.

Markdown is a content projection, not a byte-preserving HTTP archive. Preserve raw responses separately when archival requirements demand them.

Handle interaction and non-HTML material deliberately

The --eval option can execute page expressions, but is not a substitute for a complete multi-step workflow. When combined with a dump, the expression result does not automatically become the exported body. A failed expression is not evidence of a legitimate empty dataset.

For RSS, JSON, PDF, or binary resources, identify the response type first. Use an appropriate download or parser rather than forcing everything through Markdown. A direct file download may look like:

curl --fail --location --output attachment.pdf   'https://example.16yun.cn/files/manual.pdf'

The attachment URL is a placeholder. Replace it with a real resource and check content type, signature, and contents. An extension alone cannot identify the returned data.

Troubleshoot

Empty file: inspect the page log and exit status. Separate command failure from a successful command producing unusable material.

Missing dynamic fields: use a relevant readiness marker. Complex authentication and repeated interaction may justify a dedicated browser workflow.

A field exists in the semantic tree but not Markdown: inspect the projection's loss of context and choose suitable material rather than assuming Markdown preserves every page element.

Incomplete batch: reconcile accepted and failed manifests before recollecting specific inputs. Use a new batch directory and preserve provenance when combining outputs.

Screenshot or coordinate issues: check layout capability and readiness. Successful text extraction does not establish correct rendering.

Command-line composition remains dependable only when sources, accepted material, and failures remain distinguishable.