Dark Mode

author:  zep
updated: 2023-05-07 (WIP!)

▨ POD Concept

A POD ("Picotron Object Data") is a string that encodes Lua values: tables, userdata, strings, numbers booleans, and nested tables containing those types.

PODs form the basis of all data transfer and storage in Picotron. Every file is a single POD on disk, the contents of the clipboard is a POD, images embedded in documents are PODs, and messages sent between processes are PODs.

▨ POD Formats

PODs can be created with pod(val, flags). Flags indicates which encoding(s) to use:

  0x1 pxu: encode userdata in a compressed (RLE-style) form
  0x2 lz4: binary compression pass (dictionary matching)
  0x4 base64 text encoding

Any output from pod() can be converted back to the original value using unpod(pod_str)

■ Plaintext (flags 0x0)

The simplest form a POD can take is a Lua snippet that constructs the value being encoded:


This form is useful when manually constructing POD strings, or for debugging.

■ Compressed Binary (flags 0x3)

Plaintext PODs can get quite large if they contain images or map data. A compressed binary encoding can be generated using flags 0x1 and 0x2, which are normally used together as the pxu format aims to produce output that can be further compressed by lz4.

The resulting string contains non-printable characters and starts with the header "lz4\0", so only the first 3 characters are printed here:

?pod({a=1,b=2}, 0x3)

store() uses this format by default, which gives compression ratios similar to png / px9, but is faster to encode and decode.

■ Compressed Binary, Encoded as Base-64 Plaintext (flags 0x7)

The output from the binary compression pass can be returned to a text-friendly format with 0x4.

This form is still relatively compact but can be safely shared via channels like email, forums, and pasted into regular text documents. It is also used for embedding images in picotron documents and source code. Like the other plaintext format, it is also valid lua code that returns the value being encoded.

> ?pod("the quick brown fox", 0x7)
> ?unpod("b64:bHo0ABcAAAAVAAAA8AYidGhlIHF1aWNrIGJyb3duIGZveCI=")
the quick brown fox

// For small chunks of data like this one, the base64 encoding is larger than the text it encodes because of the overhead of bookkeeping information stored at each pass.

▨ POD Metadata

POD Metadata is arbitrary information about a POD such as its author, the time it was created, or hints about what to do with the pod when it is pasted or embedded somewhere.

POD metadata (if it exists) is stored at the start of a pod string in the form --[[pod...]]

This format doubles as an identifying header, so that POD files and POD strings inserted into text documents can be identified as such. It also allows the metadata part of a pod string to be ignored by the lua interpreter, and used as a regular value in .lua source code.

■ Adding Metadata to a POD

The metadata part of the POD string can be generated by pod() by giving a table of metadata as the third argument.

> str = ?pod({x=1,y=2}, 0x0, {description="an uninteresting vector"})
> ?str
--[[pod,description=\"an uninteresting vector\"]]{x=1,y=2}

unpod() returns any decoded metadata as a second result:

> c,m = unpod(str) -- returns content and metadata

Technical notes if generating metadata strings by hand: Metadata is always a table value at the top level, but the enclosing {} are not written in the pod string. When pod_type is a member of the metadata table, it is written first (unlike the usual alphabetical ordering). Otherwise a dummy value "pod" is written first. This allows the metadata to double as an indentifying header for POD data, as in both cases the pod string starts with the string: "--[[pod".

▨ POD Files

A POD can be written to disk using store(filename, val), and retrieved using fetch(filename):

> ?store("dat.pod",{1,2,"foo"})
> tbl = fetch("dat.pod")
> foreach(tbl,print)

All files in Picotron are considered to be PODs, and are stored and fetched atomically. There is no stream-like way of interacting with files, for example to read half-way into a file. A typical way to save data is to construct a temporary table containing all data to be saved, and then to store() that table.

■ File Metadata

Like unpod(), fetch() returns any metadata as the second return value. Metadata can also be read from disk separately using fetch_metadata().

Individual metadata values can be written either using the third parameter of store(), or store_metadata(). In both cases, a table of values to write is given, and any existing metadata is preserved.

> store_metadata("dat.pod", {author="zep", foo=3})
> store_metadata("dat.pod", {foo=4}) -- author is not clobbered
> meta = fetch_metadata("dat.pod")
> for k,v in pairs(meta) do print(k..": "..v) end
modified: 2023-05-07 04:23:46
author: zep
foo: 4

fetch_metadata() and store_metadata() can also be used to handle metadata about folders, including .p64 folders. Internally, the metadata is stored in a hidden file called ".info.pod".

> mkdir foo
> store_metadata("foo", {sort_by="filename"})
> ?fetch_metadata("foo").sort_by

■ Interoperability With Host OS

When the value given to store() is a single string, pod_format="raw" is recorded in the metadata, and the contents of the string are dumped to the file. This allows it to be edited in a standard text editor outside of Picotron if desired. Conversely, when a text or binary file is loaded that has a pod_format="raw", or no header at all, the contents of the file are read as a single (possibly binary) string.

▨ POD Diffs

The output of pod() is deterministic; keys are sorted to ensure that they are written in a predictable order. Two POD strings are identical when and only when the two values they encode are identical.

This makes POD strings a useful target for create_diff() and apply_diff(), to implement undo/redo stacks:

a = {1,2,3}
b = {1, "banana", 2, 3}
d = create_diff(pod(a), pod(b))
-- reconstruct b using only a and the delta (d)
s = apply_diff(pod(a), d)
foreach(unpod(s), print)