crossbind
GitHub
← All libraries

iconv

Character-set conversion.

Install

Runs in browsers; Node.js and edge runtimes use the same package. Built for wasm32, single-threaded and multi-threaded.

shell
npm install @crossbind/port-iconv-wasm@beta
crossbind.config.js
import iconvWasm from '@crossbind/port-iconv-wasm/crossbind.config.js';
 
export default {
dependencies: [iconvWasm],
paths: { config: import.meta.url },
};
main.js
import { initNative, iconv_open } from '@crossbind/port-iconv/iconv.h';
 
await initNative();

crossbind itself comes with your bundler plugin; Browser has Vite, Webpack, Rspack and Rollup.

libiconv in your browser: write what TextEncoder cannot, read what TextDecoder refuses

TextDecoder reads the Encoding Standard's list and TextEncoder writes only UTF-8. These apps run GNU libiconv 1.19, compiled by crossbind: they repair text opened with the wrong encoding, write byte-exact Shift_JIS, GB18030 or Windows-125x files, and read ISO-2022-KR, UTF-7, EUC-TW and the rest of its 112 encodings. libiconv is LGPL-2.1-or-later. The first run downloads 1.5 MB of WebAssembly once; every app on this page shares it, and nothing is uploaded.

APP 01

Repair text that was opened with the wrong encoding

été is été whose UTF-8 bytes were read as Windows-1252. The doctor turns garbled text back into the bytes it came from, which takes a legacy encoder browsers do not have, then reads those bytes in every likely encoding and ranks what comes out. Repairs through UTF-8 are close to certain; for older code pages it shows the three likeliest readings.

GARBLED TEXT
Garble your own text first
Pick a sample or paste garbled text, then repair it.
SHOW THE CODE
src/native/mojibake_doctor.h
// src/native/mojibake_doctor.h (excerpt)
for (size_t o = 0; o < openedAs().size(); o += 1) {
const std::string& opened = openedAs()[o];
// Back to the bytes the text came from: the encoder browsers lack.
const recode::Result bytes = recode::convert(garbled, "UTF-8", opened, readLikeBrowsers(opened));
if (!bytes.ok) continue;
for (size_t s = 0; s < savedAs().size(); s += 1) {
if (savedAs()[s] == opened) continue;
// Strict: a reading exists only when every byte decodes.
recode::Result step = recode::convert(bytes.output, savedAs()[s], "UTF-8");
// ... then the same mistake undone up to three times for UTF-8
}
}
main.js
const m = await initNative();
const repairs = JSON.parse(await m.MojibakeDoctor.repair('été', 3));
repairs[0].text; // 'été'
repairs[0].steps; // [{ saved: 'UTF-8', opened: 'CP1252' }]
APP 02

Write the exact bytes a legacy system imports

Banks, ERPs and government systems still import Shift_JIS, GB18030, Big5, EUC-KR or Windows-125x files, and TextEncoder only writes UTF-8. This exporter writes them byte for byte in this tab, names every character the target cannot hold, and pads fixed-width fields by bytes without splitting a character.

ROWS, ONE PER LINE

For Windows software in Japan pick CP932: JIS Shift_JIS has no ① and reads byte 5C as ¥, not a backslash.

Export the rows to see their bytes, what the encoding cannot hold, and the file.
SHOW THE CODE
src/native/legacy_export.h
// src/native/legacy_export.h (excerpt): one fixed-width field
for (size_t at = 0; at < value.size(); column += 1) {
const std::string character = value.substr(at, text::sequenceLength(static_cast<unsigned char>(value[at])));
at += character.size();
std::string bytes; // this one character in the target encoding
if (!encodeOne(cd, character, line, column, out, bytes)) {
shown.emplace_back(character, "");
continue;
}
if (cut || written.size() + bytes.size() > width) {
cut = true; // whole characters only
continue;
}
written += bytes;
shown.emplace_back(character, text::hex(bytes, bytes.size()));
}
main.js
const m = await initNative();
const rows = '山田太郎,ﾔﾏﾀﾞ ﾀﾛｳ,03-1234-5678\n';
const report = JSON.parse(await m.LegacyExport.report(rows, 'CP932', '', '8,10,12'));
// report.bytes 32, report.problems [], report.backslashes []
const file = await m.LegacyExport.file(rows, 'CP932', 'TRANSLIT', '8,10,12');
const bytes = Uint8Array.from(file, (c) => c.charCodeAt(0));
APP 03

Read the encodings TextDecoder refuses

TextDecoder reads the Encoding Standard's list and throws a RangeError for anything else, including ISO-2022-KR, ISO-2022-CN and HZ, which the standard maps to its replacement encoding on purpose. libiconv reads those too, plus UTF-7, UTF-32, EUC-TW and old DOS and Mac code pages, strictly, with the byte where decoding stops.

A file stays in this tab: it is mounted into the module's in-memory filesystem, never uploaded. Only its first 64 KB are tried; a single-byte encoding accepts almost any bytes, so read the previews.

Read the samples to compare this tab's TextDecoder with libiconv, or open a file to see which encodings can read it.
SHOW THE CODE
src/native/file_decoder.h
// src/native/file_decoder.h (excerpt)
// Strict, except that a character cut off by the read limit is not held against the file.
static recode::Result read(const Loaded& file, const std::string& encoding) {
recode::Result result = recode::convert(file.bytes, encoding, "UTF-8");
if (!result.ok && result.error == EINVAL && file.bytes.size() < file.size) result.ok = true;
return result;
}
main.js
const m = await initNative();
const [path] = await m.autoMountFiles([file], await m.getRandomPath('/memfs')); // the dropped file
const scan = JSON.parse(await m.FileDecoder.scan(path, 65536));
// scan: [{ encoding: 'CP932', ok: true, at: 46, preview: '注文番号,品名' }, ...]
const read = JSON.parse(await m.FileDecoder.decode(path, 'CP932', 16384));
// read.text, or read.error 'invalid' with read.at, read.line, read.column

Examples

Each one is a small C++ header and the JavaScript that calls it, with what it printed when the site was built. The JavaScript only tab does the same with iconv's own headers and no C++ file. The last is a whole C++ program, built for WASI.

Convert between UTF-8 and a legacy encoding

What most iconv code does: iconv_open, iconv in a loop and iconv_close, strict in both directions. A last call without input ends stateful encodings such as ISO-2022-JP, which is where the closing ESC ( B comes from.

src/native/charset.h
#pragma once
 
#include <iconv.h>
 
#include <cerrno>
#include <cstdio>
#include <memory>
#include <stdexcept>
#include <string>
#include <vector>
 
// Strict conversion between UTF-8 text and the bytes of any encoding libiconv knows. Bytes cross the
// binding as a byte string: one UTF-16 code unit (0-255) per byte.
class Charset {
public:
static std::u16string encode(const std::string& text, const std::string& encoding) {
const std::string bytes = convert(text, "UTF-8", encoding);
std::u16string units(bytes.size(), u'\0');
for (size_t i = 0; i < bytes.size(); ++i) units[i] = static_cast<unsigned char>(bytes[i]);
return units;
}
 
static std::string decode(const std::u16string& bytes, const std::string& encoding) {
std::string input(bytes.size(), '\0');
for (size_t i = 0; i < bytes.size(); ++i) {
if (bytes[i] > 0xFF) throw std::invalid_argument("not a byte string");
input[i] = static_cast<char>(bytes[i]);
}
return convert(input, encoding, "UTF-8");
}
 
private:
static std::string convert(const std::string& input, const std::string& from, const std::string& to) {
const iconv_t opened = iconv_open(to.c_str(), from.c_str());
if (opened == reinterpret_cast<iconv_t>(-1)) throw std::invalid_argument("iconv cannot convert " + from + " to " + to);
std::unique_ptr<void, int (*)(iconv_t)> cd(opened, iconv_close);
std::string output;
std::vector<char> buffer(4096);
char* in = const_cast<char*>(input.data());
size_t inLeft = input.size();
for (;;) {
char* out = buffer.data();
size_t outLeft = buffer.size();
// Once the input is used up, a call without input ends a stateful encoding (ISO-2022-JP
// switches back to ASCII); for the others it writes nothing.
const bool finishing = inLeft == 0;
const size_t status = finishing ? iconv(cd.get(), nullptr, nullptr, &out, &outLeft) : iconv(cd.get(), &in, &inLeft, &out, &outLeft);
output.append(buffer.data(), buffer.size() - outLeft);
if (status != static_cast<size_t>(-1)) {
if (finishing) return output;
} else if (errno != E2BIG) { // E2BIG only means the buffer is full, and it was just emptied
const size_t at = input.size() - inLeft;
if (errno == EINVAL) throw std::runtime_error("incomplete " + from + " input at byte " + std::to_string(at));
if (from != "UTF-8") throw std::runtime_error("invalid " + from + " input at byte " + std::to_string(at));
throw std::runtime_error(missing(input, at, to));
}
}
}
 
// Text coming from JavaScript is valid UTF-8, so EILSEQ means the target has no such character.
static std::string missing(const std::string& text, size_t at, const std::string& to) {
const auto lead = static_cast<unsigned char>(text[at]);
const size_t length = lead < 0x80 ? 1 : lead < 0xE0 ? 2 : lead < 0xF0 ? 3 : 4;
unsigned long codePoint = length == 1 ? lead : lead & (0xFF >> (length + 1));
for (size_t i = 1; i < length; ++i) codePoint = (codePoint << 6) | (static_cast<unsigned char>(text[at + i]) & 0x3F);
size_t character = 0;
for (size_t i = 0; i < at; ++i) character += (static_cast<unsigned char>(text[i]) & 0xC0) != 0x80;
char name[16];
std::snprintf(name, sizeof name, "U+%04lX", codePoint);
return "cannot encode " + std::string(name) + " at character " + std::to_string(character) + " in " + to;
}
};
main.js
import { initNative, Charset } from './native/charset.h';
 
await initNative();
const hex = (bytes) => [...bytes].map((c) => c.charCodeAt(0).toString(16).padStart(2, '0')).join(' ');
const sjis = await Charset.encode('日本語', 'SHIFT_JIS');
console.log(hex(sjis));
console.log(await Charset.decode(sjis, 'SHIFT_JIS'));
console.log(hex(await Charset.encode('日本語', 'ISO-2022-JP')));
try {
await Charset.encode('Total: €5', 'ISO-8859-1');
} catch (error) {
console.log(error.cppMessage ?? error.message);
}
PRINTSfirst run downloads 1.5 MB
93 fa 96 7b 8c ea
日本語
1b 24 42 46 7c 4b 5c 38 6c 1b 28 42
cannot encode U+20AC at character 7 in ISO-8859-1

Transliterate or drop what the target cannot hold

Append //TRANSLIT to the target and iconv approximates a missing character; append //IGNORE and it drops it. The return value of iconv counts those characters. GNU libiconv transliterates the same way in every locale, so ASCII keeps accents as marks.

src/native/legacy_encoder.h
#pragma once
 
#include <iconv.h>
 
#include <cerrno>
#include <memory>
#include <stdexcept>
#include <string>
 
// Encodes UTF-8 text for a target named the way iconv_open takes it: "ISO-8859-1" stops at the first
// character the encoding lacks, "ISO-8859-1//TRANSLIT" approximates it (€ becomes EUR) and
// "ISO-8859-1//IGNORE" drops it. The descriptor is opened once and reused for every call.
class LegacyEncoder {
public:
explicit LegacyEncoder(const std::string& target) : cd(open(target), iconv_close), target(target) {}
 
std::u16string encode(const std::string& text) {
// iconv returns how many characters it changed only when a call succeeds, so the whole text
// goes in one call, into a buffer that grows until it fits.
for (size_t capacity = text.size() * 2 + 16;; capacity *= 2) {
iconv(cd.get(), nullptr, nullptr, nullptr, nullptr); // back to the initial state
std::string output(capacity, '\0');
char* in = const_cast<char*>(text.data());
size_t inLeft = text.size();
char* out = &output[0];
size_t outLeft = capacity;
const size_t status = iconv(cd.get(), &in, &inLeft, &out, &outLeft);
if (status == static_cast<size_t>(-1) && errno != E2BIG) throw std::runtime_error(missing(text, text.size() - inLeft));
if (status == static_cast<size_t>(-1) || iconv(cd.get(), nullptr, nullptr, &out, &outLeft) == static_cast<size_t>(-1)) continue;
changes = status;
std::u16string bytes(capacity - outLeft, u'\0');
for (size_t i = 0; i < bytes.size(); ++i) bytes[i] = static_cast<unsigned char>(output[i]);
return bytes;
}
}
 
// How many characters the last encode approximated or dropped.
int changed() const { return static_cast<int>(changes); }
 
private:
static iconv_t open(const std::string& target) {
const iconv_t cd = iconv_open(target.c_str(), "UTF-8");
if (cd == reinterpret_cast<iconv_t>(-1)) throw std::invalid_argument("iconv cannot encode to " + target);
return cd;
}
 
std::string missing(const std::string& text, size_t at) const {
const auto lead = static_cast<unsigned char>(text[at]);
const size_t length = lead < 0x80 ? 1 : lead < 0xE0 ? 2 : lead < 0xF0 ? 3 : 4;
size_t character = 0;
for (size_t i = 0; i < at; ++i) character += (static_cast<unsigned char>(text[i]) & 0xC0) != 0x80;
return target + " has no \"" + text.substr(at, length) + "\" (character " + std::to_string(character) + ")";
}
 
std::unique_ptr<void, int (*)(iconv_t)> cd;
std::string target;
size_t changes = 0;
};
main.js
import { initNative, LegacyEncoder } from './native/legacy_encoder.h';
import { Charset } from './native/charset.h';
 
await initNative();
const text = 'Crème brûlée – 5 € “spécial”';
for (const target of ['ISO-8859-1//TRANSLIT', 'ISO-8859-1//IGNORE', 'ASCII//TRANSLIT']) {
const encoder = await new LegacyEncoder(target);
const bytes = await encoder.encode(text);
console.log(`${target}: ${await Charset.decode(bytes, 'ISO-8859-1')} (${await encoder.changed()} changed)`);
}
PRINTSfirst run downloads 1.5 MB
ISO-8859-1//TRANSLIT: Crème brûlée - 5 EUR "spécial" (4 changed)
ISO-8859-1//IGNORE: Crème brûlée  5  spécial (4 changed)
ASCII//TRANSLIT: Cr`eme br^ul'ee - 5 EUR "sp'ecial" (8 changed)

Decode a stream that cuts characters in two

Bytes from fetch, a WebSocket or a serial port arrive in pieces of any size, so a multibyte character can straddle two pieces. iconv stops with EINVAL at such a cut and the unfinished bytes wait for the next piece; E2BIG only means the output buffer is full.

src/native/stream_decoder.h
#pragma once
 
#include <iconv.h>
 
#include <cerrno>
#include <memory>
#include <stdexcept>
#include <string>
#include <vector>
 
// Decodes bytes that arrive in pieces (a fetch body, a WebSocket, a serial port) into UTF-8 text.
// A piece can end in the middle of a multibyte character: iconv then stops with EINVAL, and those
// bytes wait for the next piece. The calls follow Node's StringDecoder: write() every piece, then end().
class StreamDecoder {
public:
explicit StreamDecoder(const std::string& encoding) : cd(open(encoding), iconv_close), encoding(encoding) {}
 
std::string write(const std::u16string& piece) {
for (char16_t unit : piece) {
if (unit > 0xFF) throw std::invalid_argument("not a byte string");
pending += static_cast<char>(unit);
}
std::string text;
std::vector<char> buffer(4096);
char* in = &pending[0];
size_t inLeft = pending.size();
while (inLeft > 0) {
char* out = buffer.data();
size_t outLeft = buffer.size();
const size_t status = iconv(cd.get(), &in, &inLeft, &out, &outLeft);
text.append(buffer.data(), buffer.size() - outLeft);
if (status != static_cast<size_t>(-1) || errno == E2BIG) continue;
if (errno == EINVAL) { // the piece ends inside a character: keep its bytes for the next one
cuts += 1;
break;
}
throw std::runtime_error("invalid " + encoding + " input at byte " + std::to_string(position + pending.size() - inLeft));
}
position += pending.size() - inLeft;
pending.erase(0, pending.size() - inLeft);
return text;
}
 
// The stream is over: bytes still waiting are a character that never finished.
void end() {
const size_t left = pending.size();
pending.clear();
position = 0;
iconv(cd.get(), nullptr, nullptr, nullptr, nullptr);
if (left) throw std::runtime_error(encoding + " stream ends inside a character, " + std::to_string(left) + " byte(s) left over");
}
 
// How many pieces ended inside a character.
int splits() const { return cuts; }
 
private:
static iconv_t open(const std::string& encoding) {
const iconv_t cd = iconv_open("UTF-8", encoding.c_str());
if (cd == reinterpret_cast<iconv_t>(-1)) throw std::invalid_argument("iconv cannot decode " + encoding);
return cd;
}
 
std::unique_ptr<void, int (*)(iconv_t)> cd;
std::string encoding;
std::string pending;
size_t position = 0;
int cuts = 0;
};
main.js
import { initNative, StreamDecoder } from './native/stream_decoder.h';
import { Charset } from './native/charset.h';
 
await initNative();
const bytes = await Charset.encode('配送先: 東京都渋谷区神南1-2-3', 'SHIFT_JIS');
const decoder = await new StreamDecoder('SHIFT_JIS');
let text = '';
for (let at = 0; at < bytes.length; at += 3) text += await decoder.write(bytes.slice(at, at + 3));
await decoder.end();
console.log(`${bytes.length} bytes in pieces of 3, ${await decoder.splits()} characters cut in two`);
console.log(text);
 
const cut = await new StreamDecoder('SHIFT_JIS');
await cut.write(bytes.slice(0, 3));
try {
await cut.end();
} catch (error) {
console.log(error.cppMessage ?? error.message);
}
PRINTSfirst run downloads 1.5 MB
29 bytes in pieces of 3, 4 characters cut in two
配送先: 東京都渋谷区神南1-2-3
SHIFT_JIS stream ends inside a character, 1 byte(s) left over

List the encodings and check a name

iconvlist walks every encoding with its aliases and iconv_canonicalize names the canonical one, but only iconv_open proves a name works. GNU libiconv writes UTF-8 with its hyphen, and this build includes CP437 with the other DOS and EBCDIC code pages.

src/native/encodings.h
#pragma once
 
#include <iconv.h>
 
#include <string>
 
// What this build of libiconv converts. iconvlist() walks every encoding with all of its names and
// iconv_canonicalize() maps an alias to the canonical one. iconv_open() is the real test: an unknown
// name comes back from iconv_canonicalize unchanged.
class Encodings {
public:
// "1.19": _libiconv_version holds (major << 8) + minor.
static std::string version() { return std::to_string(_libiconv_version >> 8) + "." + std::to_string(_libiconv_version & 0xFF); }
 
// One entry per encoding, every name it answers to: [["CP1252","MS-ANSI","WINDOWS-1252"], ...].
static std::string list() {
std::string json = "[";
iconvlist(addEncoding, &json);
return json + "]";
}
 
// The canonical name, or "" when this build cannot convert from or to that encoding.
static std::string resolve(const std::string& name) {
const iconv_t cd = iconv_open("UTF-8", name.c_str());
if (cd == reinterpret_cast<iconv_t>(-1)) return "";
iconv_close(cd);
return iconv_canonicalize(name.c_str());
}
 
private:
static int addEncoding(unsigned int count, const char* const* names, void* data) {
std::string& json = *static_cast<std::string*>(data);
json += json.size() > 1 ? ",[" : "[";
for (unsigned int i = 0; i < count; ++i) json += std::string(i ? ",\"" : "\"") + names[i] + "\"";
json += "]";
return 0; // non-zero would stop the walk
}
};
main.js
import { initNative, Encodings } from './native/encodings.h';
 
await initNative();
const encodings = JSON.parse(await Encodings.list());
console.log(`libiconv ${await Encodings.version()}: ${encodings.length} encodings under ${encodings.flat().length} names`);
for (const name of ['latin1', 'SJIS', 'windows-1254', 'UTF8', 'CP437']) {
console.log(`${name}: ${(await Encodings.resolve(name)) || 'not supported'}`);
}
PRINTSfirst run downloads 1.5 MB
libiconv 1.19: 198 encodings under 736 names
latin1: ISO-8859-1
SJIS: SHIFT_JIS
windows-1254: CP1254
UTF8: not supported
CP437: CP437

An iconv command-line tool

WASI has no JavaScript bindings: the program is a main() that links the library, built into one .wasm and run with wasmtime. This one converts files the way the iconv command does, in 64 KB pieces, carrying a character cut between two pieces over to the next.

crossbind.config.js
import iconvWasi from '@crossbind/port-iconv-wasi/crossbind.config.js';
 
export default {
general: { name: 'iconv-tool' },
dependencies: [iconvWasi],
paths: { config: import.meta.url },
};
src/native/main.cpp
// An iconv-style converter for WASI:
// iconv-tool <from> <to> <input> <output>
// It streams the input in 64 KB pieces. A piece that ends inside a multibyte character stops iconv
// with EINVAL; those bytes wait for the next piece. A full output buffer (E2BIG) is written out and
// reused, and a last call without input ends a stateful encoding in its initial shift state.
#include <iconv.h>
 
#include <cerrno>
#include <cstdio>
#include <cstring>
#include <memory>
#include <vector>
 
namespace {
 
using File = std::unique_ptr<FILE, int (*)(FILE*)>;
 
File open(const char* path, const char* mode) {
File file(std::fopen(path, mode), std::fclose);
if (!file) std::fprintf(stderr, "cannot open %s\n", path);
return file;
}
 
bool flushOutput(FILE* file, const std::vector<char>& buffer, size_t used, long long& written) {
if (used && std::fwrite(buffer.data(), 1, used, file) != used) {
std::fprintf(stderr, "write failed\n");
return false;
}
written += static_cast<long long>(used);
return true;
}
 
} // namespace
 
int main(int argc, char** argv) {
if (argc != 5) {
std::fprintf(stderr, "usage: iconv-tool <from> <to> <input> <output>\n");
return 2;
}
const char* from = argv[1];
const char* to = argv[2];
const iconv_t opened = iconv_open(to, from);
if (opened == reinterpret_cast<iconv_t>(-1)) {
std::fprintf(stderr, "iconv cannot convert %s to %s\n", from, to);
return 1;
}
std::unique_ptr<void, int (*)(iconv_t)> cd(opened, iconv_close);
File in = open(argv[3], "rb");
File out = open(argv[4], "wb");
if (!in || !out) return 1;
 
std::vector<char> input(64 * 1024);
std::vector<char> output(64 * 1024);
size_t carried = 0;
long long read = 0;
long long written = 0;
for (;;) {
const size_t got = std::fread(input.data() + carried, 1, input.size() - carried, in.get());
if (got == 0) break;
read += static_cast<long long>(got);
char* inPointer = input.data();
size_t inLeft = carried + got;
while (inLeft > 0) {
char* outPointer = output.data();
size_t outLeft = output.size();
const size_t status = iconv(cd.get(), &inPointer, &inLeft, &outPointer, &outLeft);
if (!flushOutput(out.get(), output, output.size() - outLeft, written)) return 1;
if (status != static_cast<size_t>(-1) || errno == E2BIG) continue;
if (errno == EINVAL) break; // the piece ends inside a character: keep its bytes
std::fprintf(stderr, "invalid %s input at byte %lld\n", from, read - static_cast<long long>(inLeft));
return 1;
}
carried = inLeft;
std::memmove(input.data(), inPointer, carried);
}
if (carried) {
std::fprintf(stderr, "the %s input ends inside a character at byte %lld\n", from, read - static_cast<long long>(carried));
return 1;
}
char* outPointer = output.data();
size_t outLeft = output.size();
iconv(cd.get(), nullptr, nullptr, &outPointer, &outLeft);
if (!flushOutput(out.get(), output, output.size() - outLeft, written)) return 1;
 
std::printf("iconv %d.%d %s -> %s: %s -> %s, %lld B read, %lld B written\n", _libiconv_version >> 8, _libiconv_version & 0xFF, from, to, argv[3], argv[4], read, written);
return 0;
}
shell
npx crossbind build -p wasi -e wasi -b release
wasmtime run --dir=. .crossbind/build/iconv-tool-wasi-wasm32-st-release.wasm UTF-8 CP932 orders.csv orders-sjis.csv
wasmtime run --dir=. .crossbind/build/iconv-tool-wasi-wasm32-st-release.wasm CP932 UTF-8 orders-sjis.csv orders-copy.csv
output
iconv 1.19 UTF-8 -> CP932: orders.csv -> orders-sjis.csv, 141371 B read, 110693 B written
iconv 1.19 CP932 -> UTF-8: orders-sjis.csv -> orders-copy.csv, 110693 B read, 141371 B written
Type to search every guide page and section.
↑↓ navigate↵ openesc close