crossbind
GitHub
← All libraries

cURL

HTTP, HTTPS and file-transfer client library.

Install

Runs in browsers; Node.js and edge runtimes use the same package. Built for wasm32, single-threaded and multi-threaded.

shell
npm install @crossbind/port-curl-wasm@beta
crossbind.config.js
import curlWasm from '@crossbind/port-curl-wasm/crossbind.config.js';
 
export default {
dependencies: [curlWasm],
paths: { config: import.meta.url },
};
main.js
import { initNative, curl_url } from '@crossbind/port-curl/curl/urlapi.h';
 
await initNative();

crossbind itself comes with your bundler plugin; Browser has Vite, Webpack, Rspack and Rollup.

libcurl's own parsers in your browser, next to the browser's

In a browser this port hands curl_easy_perform to fetch, so curl's own transfers do not run here; its parsers do. These apps run libcurl 8.23.0, compiled by crossbind, beside the browser's URL and Date.parse: they find URLs that pass a JavaScript allowlist yet send libcurl to another host, take any URL apart both ways, and read HTTP dates the way curl does. The first run downloads 4.4 MB of WebAssembly once; every app on this page shares it, and nothing is uploaded.

APP 01

Test a URL allowlist against libcurl's own parser

A server that checks a URL with JavaScript's URL parser and then fetches it with libcurl runs two parsers. Where they disagree, a URL can pass the check and send libcurl to a host the check never saw. Enter the host your check allows and one it must keep out: each URL below is parsed by libcurl 8.23.0, with the flags curl_easy_perform uses, beside this browser's URL.

The check is the usual one: new URL(input).hostname must equal the allowed host. libcurl parses CURLOPT_URL with CURLU_GUESS_SCHEME and CURLU_NON_SUPPORT_SCHEME, so the host below is the one a transfer would connect to.

No request is made: these are libcurl 8.23.0 parsers running in this tab.
Run both parsers to see which URLs pass the check and where libcurl would go.
SHOW THE CODE
src/support/url_json.h
// src/support/url_json.h (excerpt)
// The flags libcurl 8.22.0 itself parses CURLOPT_URL with (lib/url.c).
constexpr unsigned int TRANSFER = CURLU_GUESS_SCHEME | CURLU_NON_SUPPORT_SCHEME;
 
CURLU* handle = curl_url();
CURLUcode code = curl_url_set(handle, CURLUPART_URL, url.c_str(), TRANSFER);
if (code != CURLUE_OK) return failure(code); // with curl_url_strerror(code)
 
char* host = nullptr;
curl_url_get(handle, CURLUPART_HOST, &host, 0);
main.js
const m = await initNative();
const input = 'http://example.com\\@attacker.example/';
 
// The check a server runs before it hands the URL to libcurl:
new URL(input).hostname === 'example.com'; // true
 
// Where libcurl will connect:
const parts = JSON.parse(await m.UrlLab.parse(input));
// parts.host: 'attacker.example', parts.user: 'example.com\\'
APP 02

Take a URL apart the way libcurl does, beside your browser

Paste a URL, and a Location header to follow if you like, to see every part libcurl reads next to what this browser's URL reads. Parts that differ are marked. The examples are known differences: backslashes, capitals and default ports, host names in other scripts, IPv4 shorthand, zone ids and redirects.

libcurl parses the URL with the flags it uses for CURLOPT_URL, CURLU_GUESS_SCHEME and CURLU_NON_SUPPORT_SCHEME, and a Location header with the ones it uses when it follows a redirect, CURLU_URLENCODE and CURLU_ALLOW_SPACE.

No request is made: these are libcurl 8.23.0 parsers running in this tab.
Parse a URL to see each part as libcurl and this browser read it.
SHOW THE CODE
src/support/url_json.h
// src/support/url_json.h (excerpt)
// The flags libcurl 8.22.0 passes itself: lib/url.c for CURLOPT_URL,
// lib/http.c for a Location header it follows.
constexpr unsigned int TRANSFER = CURLU_GUESS_SCHEME | CURLU_NON_SUPPORT_SCHEME;
constexpr unsigned int REDIRECT = CURLU_URLENCODE | CURLU_ALLOW_SPACE;
 
CURLUcode code = curl_url_set(handle, CURLUPART_URL, url.c_str(), TRANSFER);
if (code == CURLUE_OK && !reference.empty())
code = curl_url_set(handle, CURLUPART_URL, reference.c_str(), REDIRECT);
 
// Then one curl_url_get per part; CURLUE_NO_QUERY and the like become null.
char* value = nullptr;
curl_url_get(handle, CURLUPART_PORT, &value, CURLU_DEFAULT_PORT);
main.js
const m = await initNative();
const parts = JSON.parse(await m.UrlLab.parse('HTTPS://EXAMPLE.COM:443/'));
// parts.url: 'https://EXAMPLE.COM:443/', parts.host: 'EXAMPLE.COM', parts.port: '443'
new URL('HTTPS://EXAMPLE.COM:443/').href; // 'https://example.com/'
 
const next = JSON.parse(await m.UrlLab.follow('https://example.com/a/b/c', '..\\evil'));
// next.url: 'https://example.com/a/b/..\\evil'
new URL('..\\evil', 'https://example.com/a/b/c').href; // 'https://example.com/a/evil'
APP 03

Read HTTP dates the way curl does

curl_getdate is how libcurl reads Last-Modified, Expires and cookie dates. Paste dates, one per line, to see what it makes of each next to this browser's Date.parse. They part ways on dates without a zone, on two-digit years such as 70, on zone names Date.parse does not know, and on ISO 8601.

No request is made: these are libcurl 8.23.0 parsers running in this tab.
Read the dates to see each one through curl_getdate and through Date.parse.
SHOW THE CODE
src/native/http_dates.h
// src/native/http_dates.h (excerpt)
// Seconds since 1970-01-01 UTC, or -1 when curl cannot read the text as a date.
static double seconds(const std::string& text) {
return static_cast<double>(curl_getdate(text.c_str(), nullptr));
}
main.js
const m = await initNative();
await m.HttpDates.seconds('Sun Nov 6 08:49:37 1994'); // 784111777: GMT
Date.parse('Sun Nov 6 08:49:37 1994') / 1000; // local time
 
await m.HttpDates.iso('Thu, 01-Jan-70 00:00:01 GMT'); // '2070-01-01T00:00:01Z'

Examples

Each one is a small C++ header and the JavaScript that calls it, with what it printed when the site was built. The JavaScript only tab does the same with cURL's own headers and no C++ file. The last is a whole C++ program, built for WASI.

Parse a URL the way curl will fetch it

libcurl's URL API: curl_url_set checks a URL and removes its dot segments, curl_url_get reads one part back, with CURLU_DEFAULT_PORT and CURLU_URLDECODE when asked, and curl_url_strerror explains a URL curl refuses. The host keeps the case it was written in.

src/native/url_parts.h
#pragma once
 
#include <curl/curl.h>
 
#include <stdexcept>
#include <string>
 
// libcurl's URL API, the parser curl runs on a URL before every transfer. Each call parses the URL
// into a fresh CURLU handle, reads one part back and frees the handle.
class UrlParts {
public:
// The URL as curl stores it: the scheme lowercased and dot segments removed. The host keeps its case.
static std::string normalize(const std::string& url) { return get(url, CURLUPART_URL, 0); }
 
static std::string host(const std::string& url) { return get(url, CURLUPART_HOST, 0); }
 
// CURLU_DEFAULT_PORT answers with the scheme's port when the URL does not name one.
static std::string port(const std::string& url) { return get(url, CURLUPART_PORT, CURLU_DEFAULT_PORT); }
 
static std::string path(const std::string& url) { return get(url, CURLUPART_PATH, 0); }
 
// CURLU_URLDECODE turns %XX sequences back into the bytes they stand for.
static std::string query(const std::string& url) { return get(url, CURLUPART_QUERY, CURLU_URLDECODE); }
 
private:
static std::string get(const std::string& url, CURLUPart part, unsigned int flags) {
CURLU* handle = curl_url();
if (!handle) throw std::runtime_error("out of memory");
char* value = nullptr;
CURLUcode code = curl_url_set(handle, CURLUPART_URL, url.c_str(), 0);
if (code == CURLUE_OK) code = curl_url_get(handle, part, &value, flags);
curl_url_cleanup(handle);
if (code != CURLUE_OK) throw std::runtime_error(curl_url_strerror(code));
const std::string result = value;
curl_free(value);
return result;
}
};
main.js
import { initNative, UrlParts } from './native/url_parts.h';
 
await initNative();
const url = 'HTTPS://Example.com/docs/../api/search?q=caf%C3%A9#results';
console.log(await UrlParts.normalize(url));
console.log(await UrlParts.host(url), await UrlParts.port(url), await UrlParts.path(url));
console.log(await UrlParts.query(url));
try {
await UrlParts.host('https://example.com:99999/');
} catch (error) {
console.log(error.message);
}
PRINTSfirst run downloads 4.4 MB
https://Example.com/api/search?q=caf%C3%A9#results
Example.com 443 /api/search
q=café
std::runtime_error: Port number was not a decimal number between 0 and 65535

Build a URL from parts without breaking it

curl_url_set with CURLU_APPENDQUERY and CURLU_URLENCODE adds a query pair with its value escaped, CURLU_URLENCODE on the path escapes a file name, and a relative reference set on a URL is resolved the way curl follows a redirect.

src/native/url_builder.h
#pragma once
 
#include <curl/curl.h>
 
#include <stdexcept>
#include <string>
 
// Builds URLs with libcurl's URL API instead of string concatenation: references are resolved the
// way curl follows a redirect, and CURLU_URLENCODE escapes what a path or a query cannot hold.
class UrlBuilder {
public:
// `reference` is resolved against `base`, as curl does with a Location header.
static std::string resolve(const std::string& base, const std::string& reference) { return edit(base, CURLUPART_URL, reference, 0); }
 
// Replaces the path, percent-encoding spaces and other characters a path cannot hold.
static std::string withPath(const std::string& url, const std::string& path) { return edit(url, CURLUPART_PATH, path, CURLU_URLENCODE); }
 
// Appends one name=value pair to the query. The value is encoded: a space becomes +, and a
// literal + or & becomes %2B or %26.
static std::string addQuery(const std::string& url, const std::string& pair) {
return edit(url, CURLUPART_QUERY, pair, CURLU_APPENDQUERY | CURLU_URLENCODE);
}
 
private:
static std::string edit(const std::string& url, CURLUPart part, const std::string& value, unsigned int flags) {
CURLU* handle = curl_url();
if (!handle) throw std::runtime_error("out of memory");
char* result = nullptr;
CURLUcode code = curl_url_set(handle, CURLUPART_URL, url.c_str(), 0);
if (code == CURLUE_OK) code = curl_url_set(handle, part, value.c_str(), flags);
if (code == CURLUE_OK) code = curl_url_get(handle, CURLUPART_URL, &result, 0);
curl_url_cleanup(handle);
if (code != CURLUE_OK) throw std::runtime_error(curl_url_strerror(code));
const std::string text = result;
curl_free(result);
return text;
}
};
main.js
import { initNative, UrlBuilder } from './native/url_builder.h';
 
await initNative();
let url = await UrlBuilder.resolve('https://api.example.com/v1/', 'search');
url = await UrlBuilder.addQuery(url, 'q=crème brûlée & tea');
url = await UrlBuilder.addQuery(url, 'sort=price+asc');
console.log(url);
console.log(await UrlBuilder.resolve(url, '../v2/items?id=7'));
console.log(await UrlBuilder.withPath(url, '/files/Q3 report.pdf'));
PRINTSfirst run downloads 4.4 MB
https://api.example.com/v1/search?q=cr%C3%A8me+br%C3%BBl%C3%A9e+%26+tea&sort=price%2Basc
https://api.example.com/v2/items?id=7
https://api.example.com/files/Q3%20report.pdf?q=cr%C3%A8me+br%C3%BBl%C3%A9e+%26+tea&sort=price%2Basc

Percent-encode and decode text

curl_easy_escape encodes every byte except letters, digits and - . _ ~, the encoding the curl tool's --data-urlencode and --url-query use. JavaScript's encodeURIComponent leaves ( ) ! * ' alone. curl_easy_unescape decodes %XX only: + stays +, and a % that starts no valid sequence is kept.

src/native/percent_codec.h
#pragma once
 
#include <curl/curl.h>
 
#include <cstddef>
#include <cstring>
#include <stdexcept>
#include <string>
 
// curl_easy_escape and curl_easy_unescape: percent-encoding the way curl does it. Every byte except
// A-Z, a-z, 0-9 and - . _ ~ is escaped, so the result is safe in any part of a URL. Since libcurl
// 7.82.0 both functions ignore their handle argument, so no easy handle is needed.
class PercentCodec {
public:
static std::string escape(const std::string& text) {
char* escaped = curl_easy_escape(nullptr, text.data(), static_cast<int>(text.size()));
return take(escaped, escaped ? std::strlen(escaped) : 0);
}
 
// Decodes %XX sequences only: + stays +, and a % that starts no valid sequence is kept.
static std::string unescape(const std::string& text) {
int size = 0;
char* decoded = curl_easy_unescape(nullptr, text.data(), static_cast<int>(text.size()), &size);
return take(decoded, static_cast<std::size_t>(size));
}
 
private:
static std::string take(char* data, std::size_t size) {
if (!data) throw std::runtime_error("out of memory");
const std::string result(data, size);
curl_free(data);
return result;
}
};
main.js
import { initNative, PercentCodec } from './native/percent_codec.h';
 
await initNative();
const text = 'crème brûlée & tea (50%)';
const escaped = await PercentCodec.escape(text);
console.log(escaped);
console.log(await PercentCodec.unescape(escaped));
console.log(encodeURIComponent(text));
console.log(await PercentCodec.unescape('a+b%20c%zz'));
PRINTSfirst run downloads 4.4 MB
cr%C3%A8me%20br%C3%BBl%C3%A9e%20%26%20tea%20%2850%25%29
crème brûlée & tea (50%)
cr%C3%A8me%20br%C3%BBl%C3%A9e%20%26%20tea%20(50%25)
a+b c%zz

Read the dates in HTTP headers

curl_getdate reads the three date formats HTTP allows, and variations such as zone names, as seconds since 1970 in UTC. A date without a zone is GMT, where JavaScript's Date.parse takes local time, and ISO 8601 is not a format it reads.

src/native/http_dates.h
#pragma once
 
#include <curl/curl.h>
 
#include <ctime>
#include <string>
 
// curl_getdate, the date parser curl uses for Last-Modified, Expires, cookie expiry and the tool's
// -z option. It reads the three HTTP date formats and many variations, and a date without a time
// zone is taken as GMT.
class HttpDates {
public:
// Seconds since 1970-01-01 UTC, or -1 when curl cannot read the text as a date.
static double seconds(const std::string& text) { return static_cast<double>(curl_getdate(text.c_str(), nullptr)); }
 
// The same instant written as ISO 8601 in UTC.
static std::string iso(const std::string& text) {
const time_t when = curl_getdate(text.c_str(), nullptr);
if (when == -1) return "not a date";
std::tm parts{};
gmtime_r(&when, &parts);
char buffer[32];
std::strftime(buffer, sizeof buffer, "%Y-%m-%dT%H:%M:%SZ", &parts);
return buffer;
}
};
main.js
import { initNative, HttpDates } from './native/http_dates.h';
 
await initNative();
const formats = ['Sun, 06 Nov 1994 08:49:37 GMT', 'Sunday, 06-Nov-94 08:49:37 GMT', 'Sun Nov 6 08:49:37 1994'];
for (const date of formats) {
console.log(await HttpDates.seconds(date), await HttpDates.iso(date));
}
console.log(await HttpDates.iso('Sun, 06 Nov 1994 08:49:37 CEST'));
console.log(await HttpDates.seconds('1994-11-06T08:49:37Z'));
PRINTSfirst run downloads 4.4 MB
784111777 1994-11-06T08:49:37Z
784111777 1994-11-06T08:49:37Z
784111777 1994-11-06T08:49:37Z
1994-11-06T06:49:37Z
-1

Check what this libcurl was built with

curl_version_info lists the protocols and features compiled in, so code can check before relying on one. This is the WebAssembly build: no HTTP/2, no compression, no IDN and no IPv6. Its protocols are compiled in, but in a browser a transfer goes through fetch, so only HTTP and HTTPS leave the page. The Android and iOS packages report the same protocols, plus libz.

src/native/build_info.h
#pragma once
 
#include <curl/curl.h>
 
#include <string>
 
// curl_version_info reports what this libcurl was built with, so code can check for a protocol or
// a feature before relying on it.
class BuildInfo {
public:
static std::string version() { return curl_version(); }
 
static std::string protocols() { return join(info()->protocols); }
 
static std::string features() { return join(info()->feature_names); }
 
static bool supports(const std::string& feature) {
for (const char* const* item = info()->feature_names; item && *item; ++item) {
if (feature == *item) return true;
}
return false;
}
 
private:
static const curl_version_info_data* info() { return curl_version_info(CURLVERSION_NOW); }
 
static std::string join(const char* const* items) {
std::string text;
for (; items && *items; ++items) text += (text.empty() ? "" : " ") + std::string(*items);
return text;
}
};
main.js
import { initNative, BuildInfo } from './native/build_info.h';
 
await initNative();
console.log(await BuildInfo.version());
console.log(await BuildInfo.protocols());
console.log(await BuildInfo.features());
console.log(await BuildInfo.supports('HTTP2'), await BuildInfo.supports('HSTS'));
PRINTSfirst run downloads 4.4 MB
libcurl/8.22.0 OpenSSL/4.0.3
dict file ftp ftps gopher gophers http https imap imaps mqtt mqtts pop3 pop3s rtsp smtp smtps telnet tftp ws wss
alt-svc AsynchDNS HSTS HTTPS-proxy Largefile SSL threadsafe UnixSockets
false true

A URL tool on libcurl, and the curl command itself

WASI has no JavaScript bindings: the program is a main() that links libcurl, built into one .wasm and run with wasmtime. This one is a small trurl: it parses a URL with the flags curl uses before a transfer, follows a Location header and appends encoded query pairs, without opening a connection. For transfers, @crossbind/port-curl-standalone-wasi ships the curl command itself, which fetches over wasi:sockets with OpenSSL and verifies certificates: npx -p @crossbind/port-curl-standalone-wasi@beta curl-wasi -sS https://example.com -o page.html.

crossbind.config.js
import curlWasi from '@crossbind/port-curl-wasi/crossbind.config.js';
 
export default {
general: { name: 'url-tool' },
dependencies: [curlWasi],
// A separate output folder: crossbind 2.0.0-beta.60 stops with ENOENT at the end of a WASI build
// whose dependencies ship data (OpenSSL's CA certificates here) when output is the build folder.
paths: { config: import.meta.url, output: 'dist' },
};
src/native/main.cpp
// A trurl-style URL tool on libcurl's URL API, for WASI. It parses a URL with the flags libcurl uses
// for CURLOPT_URL, then --follow resolves a Location header against it the way curl follows a
// redirect and --append adds a URL-encoded query pair. It prints the URL and its parts and opens no
// connection.
// url-tool <url> [--follow <location>] [--append <name=value>]...
#include <curl/curl.h>
 
#include <cstdio>
#include <cstring>
#include <string>
 
namespace {
 
constexpr unsigned int TRANSFER = CURLU_GUESS_SCHEME | CURLU_NON_SUPPORT_SCHEME;
constexpr unsigned int REDIRECT = CURLU_URLENCODE | CURLU_ALLOW_SPACE;
 
int usage() {
std::fprintf(stderr, "usage: url-tool <url> [--follow <location>] [--append <name=value>]...\n");
return 2;
}
 
// Adds "name=value" for a part the URL has, nothing for one it lacks.
void part(std::string& line, CURLU* url, const char* name, CURLUPart which, unsigned int flags) {
char* value = nullptr;
if (curl_url_get(url, which, &value, flags) != CURLUE_OK) return;
line += (line.empty() ? "" : " ") + std::string(name) + "=" + value;
curl_free(value);
}
 
} // namespace
 
int main(int argc, char** argv) {
if (argc < 2 || argc % 2 != 0) return usage();
for (int i = 2; i < argc; i += 2) {
if (std::strcmp(argv[i], "--follow") != 0 && std::strcmp(argv[i], "--append") != 0) return usage();
}
CURLU* url = curl_url();
if (!url) return 1;
CURLUcode code = curl_url_set(url, CURLUPART_URL, argv[1], TRANSFER);
for (int i = 2; code == CURLUE_OK && i < argc; i += 2) {
code = std::strcmp(argv[i], "--follow") == 0 ? curl_url_set(url, CURLUPART_URL, argv[i + 1], REDIRECT)
: curl_url_set(url, CURLUPART_QUERY, argv[i + 1], CURLU_APPENDQUERY | CURLU_URLENCODE);
}
char* whole = nullptr;
if (code == CURLUE_OK) code = curl_url_get(url, CURLUPART_URL, &whole, 0);
if (code != CURLUE_OK) {
std::fprintf(stderr, "url-tool: %s\n", curl_url_strerror(code));
curl_url_cleanup(url);
return 1;
}
std::printf("%s\n", whole);
curl_free(whole);
std::string parts;
part(parts, url, "scheme", CURLUPART_SCHEME, 0);
part(parts, url, "user", CURLUPART_USER, 0);
part(parts, url, "host", CURLUPART_HOST, 0);
part(parts, url, "port", CURLUPART_PORT, CURLU_DEFAULT_PORT);
part(parts, url, "path", CURLUPART_PATH, 0);
part(parts, url, "query", CURLUPART_QUERY, 0);
part(parts, url, "fragment", CURLUPART_FRAGMENT, 0);
std::printf("%s\n", parts.c_str());
curl_url_cleanup(url);
return 0;
}
shell
npx crossbind build -p wasi -e wasi -b release
wasmtime run .crossbind/build/url-tool-wasi-wasm32-st-release.wasm 'http://example.com\@attacker.example/'
wasmtime run .crossbind/build/url-tool-wasi-wasm32-st-release.wasm https://example.com/a/b/c --follow '../d?y=2'
wasmtime run .crossbind/build/url-tool-wasi-wasm32-st-release.wasm https://example.com/search --append 'q=crème brûlée & tea' --append 'sort=price+asc'
output
http://example.com\@attacker.example/
scheme=http user=example.com\ host=attacker.example port=80 path=/
https://example.com/a/d?y=2
scheme=https host=example.com port=443 path=/a/d query=y=2
https://example.com/search?q=cr%C3%A8me+br%C3%BBl%C3%A9e+%26+tea&sort=price%2Basc
scheme=https host=example.com port=443 path=/search query=q=cr%C3%A8me+br%C3%BBl%C3%A9e+%26+tea&sort=price%2Basc
Type to search every guide page and section.
↑↓ navigate↵ openesc close