# URL to Markdown: converter, Python, CLI or API compared

Four ways to turn a web page into Markdown: pandoc on the command line, Python libraries, a hosted API or a batch tool, with real commands, costs and limits.

By Tan Solutions. Published 15 November 2026. Updated 15 November 2026.

## Short answer

There are four ways to turn a URL into Markdown. A **command-line converter** (pandoc, or trafilatura's CLI) is fastest for one page. A **Python library** (markdownify or trafilatura) fits when the conversion lives inside your own code. A **hosted API** (for example Cloudflare's `/markdown` endpoint) adds a real browser for JavaScript pages. A **batch tool** fits when you have hundreds of URLs and need a clear result for every one, including the ones that fail.

> **Do this for a list:** [URL to Markdown](https://www.tansolutions.org/tools/url-to-markdown/) converts up to 1,000 URLs per run into clean Markdown with title and metadata, opens a browser only when a page needs it, and charges only for pages that convert: from USD 1.00 per 1,000 converted pages.

## 1. The four ways compared

Every fact below comes from the project's own documentation or package page, read on 15 November 2026.

| Feature | pandoc (CLI) | markdownify (Python) | trafilatura (Python and CLI) | Cloudflare `/markdown` (API) | URL to Markdown (ours) |
|---|---|---|---|---|---|
| Current version | 3.14, 10 Nov 2026 | 1.2.3, 30 Jun 2026 | 2.3.1, 6 Oct 2026 | hosted service | hosted on Apify |
| Fetches the URL itself | Yes, "using HTTP" | No, you pass it HTML | Yes, `fetch_url` or `-u` | Yes, in a browser | Yes; browser only when needed |
| Picks the main text | No, converts what it gets | No; you choose tags with `strip` or `convert` | Yes; `html2txt()` gives all text | not stated on the endpoint page | `extractMode`: `full` or `article` |
| JavaScript pages | No browser | No browser | No browser | Yes, with a section on JS-heavy pages | Yes, `renderMode` `auto` or `browser` |
| Metadata | No | No | Yes, `with_metadata=True` | not stated on the endpoint page | Title, description, language, dates, canonical URL |
| Many URLs | Your own loop | Your own loop | Built-in crawling options | One URL or HTML per call | Up to 1,000 per run, status per URL |
| Cost | Free, runs locally | Free, runs locally | Free, runs locally | Browser time (section 5) | Per converted page (section 5) |

The local tools are the right answer more often than not. The hosted options earn their keep on JavaScript-heavy sites and on long lists where you need to know exactly which pages failed and why.

## 2. On the command line

**pandoc** reads straight from the web. Its manual says: "Instead of an input file, an absolute URI may be given. In this case pandoc will fetch the content using HTTP" ([pandoc manual](https://pandoc.org/MANUAL.html#reading-from-the-web), read 15 November 2026):

```bash
pandoc -f html -t markdown https://go.dev/doc/tutorial/getting-started
pandoc -f html -t gfm -o tutorial.md https://go.dev/doc/tutorial/getting-started
```

The second line writes GitHub-flavoured Markdown to a file. pandoc converts the whole document, so menus and footers come along, and it does not run JavaScript.

**trafilatura** has a CLI that keeps the main content only. The `-u` option takes a URL, and `--markdown` (or `--output-format markdown`) selects Markdown ([trafilatura docs](https://trafilatura.readthedocs.io/en/latest/usage-cli.html), read 15 November 2026):

```bash
pip install trafilatura
trafilatura -u "https://go.dev/doc/tutorial/getting-started" --markdown
```

## 3. In Python

**markdownify** converts an HTML string; fetching the page is up to you. Its default heading style is `UNDERLINED`, so set `ATX` if you want `#` headings ([PyPI](https://pypi.org/project/markdownify/), read 15 November 2026):

```python
import requests
from markdownify import markdownify as md

html = requests.get("https://go.dev/doc/tutorial/getting-started", timeout=30).text
markdown = md(html, heading_style="ATX", strip=["script", "style"])
print(markdown[:500])
```

This gives you exact control over each tag and keeps everything on the page, navigation included. Cut the HTML down to the `<main>` or `<article>` element first if you only want the content.

**trafilatura** fetches, picks the main content and returns Markdown with metadata in one call ([quickstart](https://trafilatura.readthedocs.io/en/latest/quickstart.html), read 15 November 2026):

```python
from trafilatura import fetch_url, extract

downloaded = fetch_url("https://go.dev/doc/tutorial/getting-started")
markdown = extract(downloaded, output_format="markdown", with_metadata=True)
print(markdown[:500])
```

Its documentation explains that `extract()` runs "a cascade of extractors (its own rules, then readability and jusText as fallbacks)", and that `fast=True` skips the fallbacks, "making extraction roughly twice as fast at the cost of potentially missing content on difficult pages".

## 4. Through an API

When pages build their text with JavaScript, a plain HTTP fetch returns a nearly empty shell. A hosted browser solves that. Cloudflare's Browser Run has a `/markdown` endpoint that "retrieves a webpage's content and converts it into Markdown format". You send `url` (or raw `html`) with an API token, and the answer is JSON with the Markdown in `result` ([Cloudflare docs](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/), last updated 26 September 2026, read 15 November 2026):

```bash
curl -X POST "https://api.cloudflare.com/client/v4/accounts/<accountId>/browser-run/markdown" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <apiToken>" \
  -d '{"url": "https://example.com"}'
```

You need a Cloudflare account and a token with the "Browser Run - Edit" permission. The endpoint takes one URL or one HTML string per call; looping over a list, retries and a per-page failure report are yours to build.

## 5. What it costs per 1,000 pages

- **pandoc, markdownify, trafilatura:** free to install; you pay for your own machine and time.
- **Cloudflare Browser Run:** billed by browser time, not pages. The pricing page lists 10 minutes per day on Workers Free, and 10 hours per month on Workers Paid, "then" USD 0.09 per additional hour (last updated 21 April 2026, read 15 November 2026). Your cost per 1,000 pages depends on how long each page keeps the browser busy; measure a sample before you estimate.
- **URL to Markdown (ours):** from USD 1.00 per 1,000 converted pages, plus Apify's own charge for starting a run. It is USD 1.50 per 1,000 converted pages on the Apify free plan. Pages that fail or time out, pages a bot wall or CAPTCHA blocks, pages robots.txt does not allow, URLs over your cap and pages too big for the run memory are free.

## 6. What clean Markdown keeps

For a search index, a chatbot or a prompt, the goal is the text a reader needs and nothing else:

- **Keep:** headings (so chunks have context), paragraphs, lists, tables, code blocks with their language, links and image references.
- **Drop:** menus, footers, cookie banners, share buttons and inline scripts.
- **Add:** the title, the canonical URL, the language and the published or modified date, so every chunk can be traced back to its page.

For retrieval-augmented generation (RAG), split long pages at headings and paragraphs rather than every N characters. A chunk that cuts a code block or a table in half retrieves badly.

## 7. Bot walls and failures

At scale, some pages will not convert, and a good pipeline says which and why. The usual causes are a timeout, an HTTP error status, a page that is empty without JavaScript, a CAPTCHA or bot wall, and a robots.txt rule that asks crawlers to stay out. pandoc lets you set any User-Agent header, but pretending to be a browser to get past a block is a choice about the site's wishes, not a technical fix. We recommend you identify your crawler and respect robots.txt.

## 8. With our tool

[URL to Markdown](https://www.tansolutions.org/tools/url-to-markdown/) reads plain HTML first and opens a real browser only when a page turns out to be an empty JavaScript shell. A real input:

```json
{
  "startUrls": [
    { "url": "https://go.dev/doc/tutorial/getting-started" },
    { "url": "https://docs.python.org/3/tutorial/introduction.html" }
  ],
  "maxUrls": 100,
  "renderMode": "auto",
  "extractMode": "full",
  "includeImages": true,
  "includeLinks": true,
  "frontMatter": false,
  "chunkSize": 2000,
  "timeoutSecs": 30,
  "maxConcurrency": 5
}
```

One result per URL, in the order you gave them. This is a real result from a test run on 13 November 2026, with the Markdown cut short (that run had no chunk size, so `chunks` is null):

```json
{
  "url": "https://go.dev/doc/tutorial/getting-started",
  "status": "ok",
  "title": "Tutorial: Get started with Go - The Go Programming Language",
  "markdown": "# Tutorial: Get started with Go\n\nIn this tutorial, you'll get a brief introduction to Go programming. ...",
  "wordCount": 1041,
  "chunks": null,
  "metadata": { "lang": "en", "canonicalUrl": "https://go.dev/doc/tutorial/getting-started" },
  "fetchedWith": "http",
  "httpStatus": 200,
  "crawledAt": "2026-11-13T07:49:36.789Z",
  "error": null
}
```

The status is `ok`, `robots_disallowed`, `blocked` or `error`. A failed result carries an error code such as `TIMEOUT`, `HTTP_ERROR` or `EMPTY_CONTENT` and a plain message. With `chunkSize` set (500 to 100,000 characters), `chunks` is a list of Markdown strings split at headings and paragraphs. `fetchedWith` shows whether the page needed the browser. Run it on the [Apify Store](https://apify.com/tan-solutions/url-to-markdown), or schedule it daily to keep a knowledge base current.

Limits: public pages only (no logins, cookies, forms or paywalls); it stops at CAPTCHAs and bot walls and does not hide who it is; up to 1,000 URLs per run and at most 5 million characters of Markdown per result. The crawler identifies itself as TanForgeBot. For PDF, Word and PowerPoint files, use [PDF, Word and PowerPoint to Markdown](https://www.tansolutions.org/tools/document-to-markdown/) instead.

## Sources

All read on 15 November 2026. Undated official pages are the current versions on that day.

1. pandoc, [User's Guide: Reading from the Web](https://pandoc.org/MANUAL.html#reading-from-the-web), no page date.
2. pandoc, [Releases](https://pandoc.org/releases.html), pandoc 3.14 of 10 November 2026.
3. PyPI, [markdownify 1.2.3](https://pypi.org/project/markdownify/), released 30 June 2026.
4. PyPI, [trafilatura 2.3.1](https://pypi.org/project/trafilatura/), released 6 October 2026.
5. Trafilatura documentation, [Quickstart](https://trafilatura.readthedocs.io/en/latest/quickstart.html) and [Command-line usage](https://trafilatura.readthedocs.io/en/latest/usage-cli.html), version 2.3.1.
6. Cloudflare, [/markdown endpoint](https://developers.cloudflare.com/browser-run/quick-actions/markdown-endpoint/), last updated 26 September 2026.
7. Cloudflare, [Browser Run pricing](https://developers.cloudflare.com/browser-run/pricing/), last updated 21 April 2026.
8. Tan Solutions, [URL to Markdown](https://apify.com/tan-solutions/url-to-markdown), test run of 13 November 2026, verified 14 November 2026.

## Tools for this task

- [URL to Markdown](https://www.tansolutions.org/tools/url-to-markdown/): Turns web pages into clean Markdown with title and metadata, ready for search indexes and knowledge bases. From USD 1.00 per 1,000 converted pages. Apify Store: https://apify.com/tan-solutions/url-to-markdown

Source: https://www.tansolutions.org/guides/url-to-markdown-converter-python-api/ · Tan Solutions
