Letterbox.Charset behaviour (Letterbox v0.1.0-rc.0)

The text of an email in UTF-8, whatever charset it was written in.

Letterbox knows UTF-8 and the single-byte encodings of the WHATWG Encoding Standard, as browsers do: Windows-1250 to 1258, the ISO-8859 family, KOI8-R and KOI8-U, Mac Roman and others, under every label WHATWG lists. Like a browser, it reads iso-8859-1 and us-ascii as Windows-1252, which they nearly always are in real email: the € and the curly quotes of Outlook's Latin-1 are bytes Latin-1 does not have.

An encoding of several bytes a character (Shift_JIS, ISO-2022-JP, GBK, Big5, EUC-KR) goes to a module of the app's, when it has one:

config :letterbox, charsets: MyApp.Charsets

defmodule MyApp.Charsets do
  @behaviour Letterbox.Charset

  @impl true
  def decode(bytes, "shift_jis"), do: MyApp.Iconv.convert(bytes, "SHIFT_JIS", "UTF-8")
  def decode(_bytes, _label), do: :error
end

What no one can read is given with each unknown byte as U+FFFD, and decode/2 says so (:lossy), so a message can tell that its text is not all there.

Summary

Callbacks

decode(bytes, label)

Decodes bytes of the encoding label (lowercase, trimmed): {:ok, utf8}, or :error for a label the module does not know.

Functions

decode(bytes, label)

bytes in UTF-8: {:ok, text} when every byte was read, {:lossy, text} when some became U+FFFD (bytes the encoding does not have, or an encoding no one knows).

known?(label)

Whether Letterbox reads label itself (UTF-8, or one of WHATWG's single-byte encodings), without the app's module.

to_utf8(bytes, label)

decode/2's text, lossy or not.

Callbacks

decode(bytes, label)

@callback decode(bytes :: binary(), label :: String.t()) :: {:ok, String.t()} | :error

Decodes bytes of the encoding label (lowercase, trimmed): {:ok, utf8}, or :error for a label the module does not know.

Functions

decode(bytes, label)

@spec decode(binary(), String.t() | nil) :: {:ok, String.t()} | {:lossy, String.t()}

bytes in UTF-8: {:ok, text} when every byte was read, {:lossy, text} when some became U+FFFD (bytes the encoding does not have, or an encoding no one knows).

known?(label)

@spec known?(String.t() | nil) :: boolean()

Whether Letterbox reads label itself (UTF-8, or one of WHATWG's single-byte encodings), without the app's module.

to_utf8(bytes, label)

@spec to_utf8(binary(), String.t() | nil) :: String.t()

decode/2's text, lossy or not.