2017-03-22: Kaitai Struct v0.7 released

Kaitai project is happy to announce release of new major version of Kaitai Struct, declarative markup language to describe various binary data structures — binary file formats, network stream packets, etc.

The basic idea of Kaitai Struct is that a particular format can be described using Kaitai Struct language (in a .ksy file), which then can be compiled using ksc into source files in one of the supported programming languages. These modules will include a generated code for a parser that can read described data structure from a file / stream and provide access to its contents in a nice, easy-to-comprehend API.

Release highlights

• New ksy features:
  • Type importing system: meta/imports can be used to import other types as first-class citizens in current compilation unit; "opaque types" are now disabled by default (see below)
  • Byte-terminated notation (terminator, include and consume) can be now used not only for strings, but also for any byte types and user types
  • pad-right to remove declare excess right padding (usually with 0s)
  • User types can now use parent: expression to enforce a specific parent for an object, or parent: false to disable parenting at all (and, subsequently, remove it from parent type inferring process)
  • Type inferring: value instances are now allowed to use _parent
  • doc-ref to add references to external documentation for types / attributes
• Improved compilation process:
  • Compilation is now clearly separated in 3 phases: YAML parsing, precompilation, compilation. Phases 1 and 2 are language-agnostic and "precompilation" now does all possible sanity checks preliminary, making sure that language-specific "compilation" doesn't have to deal with invalid data.
  • Improved compilation results reporting: now all error messages reported by compiler have file / code location and proper user-readable text. Added more than 50 tests for erroneous input files. Exceptions thrown directly are considered a compiler bug from now on.
  • Generated code now checks for runtime library version compatibility and fails to compile / run with non-compliant runtime
• Command-line compiler options:
  • --opaque-types=true to enable opaque types (disabled by default, i.e. using unknown type would be treated as error)
  • --verbose now allows fine-tuned verbose logging for various compiler's subsystems; using --verbose=all exposes a lot of internal logic.
  • --ksc-json-output to dump compilation results in machine-readable JSON format (simplifies ksc integration in other tools, like visualizers)
• Console visualizer: faster loading, automatic handling of imports (no more need to specify all .ksy files manually on invocation)
• Expression language:
  • Two string types: single quotes (verbatim), double quotes (interpolating with escape characters)
  • New type casting operator: .as<foo>
  • New methods:
    • arrays: size
    • booleans: to_i
    • byte arrays: to_s(encoding)
    • enums: to_i
    • strings: reverse
• Runtime API changes:
  • All bytearray to string functions are named bytes_to_str in all languages
  • Added read_bytes_term (akin to what read_str_term did previously to strings)
  • Removed read_str_* methods, they are to be replaced now with combination of read_bytes_* + bytes_to_str
  • Added bytes_strip_right and bytes_terminate
  • Perl module now uses IO::KaitaiStruct package name (instead of Kaitai)
• Major bugfixes:
  • Recursive top-level types
  • Unaligned bits reading with enums on top of bit-level integers
  • repeat-until handling with substreams