2016-11-09: Kaitai Struct v0.5 released

Kaitai project is happy to announce release of new major version of Kaitai Struct, declarative markup language to describe various binary data structures — binary file formats, network stream packets, etc.

The basic idea of Kaitai Struct is that a particular format can be described using Kaitai Struct language (in a .ksy file), which then can be compiled using ksc into source files in one of the supported programming languages. These modules will include a generated code for a parser that can read described data structure from a file / stream and provide access to its contents in a nice, easy-to-comprehend API.

Release highlights

  • Target languages support:
    • C++/STL - fully supported, all tests pass
    • Python - made compiled code and runtime compatible with both Python 2 and 3, enforced by CI
    • PHP7 - new target language, 98% supported
    • Perl - new target language, 85% supported
    • Graphviz - allows generation of visual diagrams of data formats, to be laid out with GraphViz (.dot format)
  • New KSY language features:
    • Switch-like conditional structure to determine type based on value of expression (instead of tons of ifs)
    • Attribute field doc to annotate fields - will generate docstrings relevant to language (i.e. JavaDoc, JSDoc, YARD/RDoc, etc)
    • repeat-until allows repetition of a field until a condition is met
    • Boolean type support
  • Expression language:
    • _io.eof returns boolean value - whether the end of stream was reached or not
    • _io.pos returns current position in the stream
    • _io.size returns size of the stream
  • .ksy parsing improvements:
    • New unified type derivation engine allows compile-time type error checks and full support of target languages which require absolute type designations (like C++, Python, Perl or PHP)
    • Same YAML parsing code is now used for both JVM and JS platforms
    • Stricter checks on all parsing stages: lots of invalid combinations are now prohibited (instead of choosing one of variants)
    • Better error messages: now in most cases compiler clearly indicates source of the problem
  • Build and release process:
    • Compiler: added building as pom module
    • Java runtime: added building as pom module
    • Python runtime: added building as pip module
    • Windows CI: now all commits are built also on Windows, with .msi package available for download
  • Debug mode:
    • Support implemented for Java and JavaScript (to allow creation of visualizer tools in these languages - see Java GUI for Kaitai Struct and Web IDE for Kaitai Struct
    • Added generation of SEQ_FIELDS helper const array that allows clear separation of sequence attributes vs instance without guesswork
    • Exception in debug mode now tries to save as much parsed data as possible (to aid diagnosing the error)
  • Incompatible changes:
    • Identifiers are now strictly checked to conform to lower_underscore_case pattern (that would be converted to language-specific style on ouput)
    • Java:
      • _parse method renamed to _read
      • process* methods are now static
    • JavaScript: position in runtime is renamed to pos (to conform to general KS API spec)
    • Compiler API: now all compilers accept unified RuntimeConfig for configuration instead of individual options
  • Bugfixes:
    • Java:
      • having if on a sequence attribute now makes it automatically boxed (to allow it to be null)
      • work around some int vs long incompatibilities
      • proper boxing of floating types
    • Integer modulo (%) operation now behaves exactly the same in all languages, always returning positive result (as opposed to remainder operation % in languages like C++ or Java)

0.5 released 2016-11-09

Download

Try in browser