Reading and writing binary formats is hard, especially if it’s an interchange format that should work across a multitude of platforms and languages.

Have you ever found yourself writing repetitive, error-prone and hard-to-debug code that reads binary data structures from files or network streams and somehow represents them in memory for easier access?

Kaitai Struct tries to make this job easier — you only have to describe the binary format once and then everybody can use it from their programming languages — cross-language, cross-platform.

What is Kaitai Struct?

Kaitai Struct is a declarative language used to describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc.

The main idea is that a particular format is described in Kaitai Struct language (.ksy file) and then can be compiled with ksc into source files in one of the supported programming languages. These modules will include a generated code for a parser that can read the described data structure from a file or stream and give access to it in a nice, easy-to-comprehend API.

Using KS in your project

Typically, using formats described in KS in your project involves the following steps:

• Describe the format — i.e. create a .ksy file
• Use a visualizer to debug the format and ensure that it parses data properly
• Compile the .ksy file into a target language source file and include that file into your project
• Add the KS runtime library for your particular language into your project (don’t worry, it’s small and it’s there mostly to ensure readability of generated code)
• Use the generated class(es) to parse your binary file or stream and access its components

Check out the documentation for more information.

meta:
  id: tcp_segment
  endian: be
seq:
  - id: src_port
    type: u2
  - id: dst_port
    type: u2
  - id: seq_num
    type: u4
  - id: ack_num
    type: u4

public class TcpSegment extends KaitaiStruct {
    // ...
    private void _read() throws IOException {
        this.srcPort = _io.readU2be();
        this.dstPort = _io.readU2be();
        this.seqNum = _io.readU4be();
        this.ackNum = _io.readU4be();
    }
    // ...

meta:
  id: gif
  file-extension: gif
  endian: le
seq:
  - id: header
    type: header
  - id: logical_screen
    type: logical_screen
types:
  header:
    seq:
      - id: magic
        contents: 'GIF'
      - id: version
        size: 3
  logical_screen:
    seq:
      - id: image_width
        type: u2
      - id: image_height
        type: u2
      - id: flags
        type: u1
      - id: bg_color_index
        type: u1
      - id: pixel_aspect_ratio
        type: u1

It declares that a GIF file usually has a .gif extension and uses little-endian integer encoding. The file itself starts with two blocks: first comes header and then comes logical_screen:

• “Header” consists of a “magic” string of 3 bytes (“GIF”) that identifies that it’s a GIF file starting and then there are 3 more bytes that identify the format version (87a or 89a).
• “Logical screen descriptor” is a block of integers:
  • image_width and image_height are 2-byte unsigned ints
  • flags, bg_color_index and pixel_aspect_ratio take 1-byte unsigned ints each

This .ksy file can be compiled into gif.cpp / Gif.cs / gif.go / Gif.java / Gif.js / gif.lua / gif.nim / Gif.pm / Gif.php / gif.py / gif.rb and then one can instantly load a .gif file and access, for example, its width and height.

• C++/STL
• C#
• Go
• Java
• JavaScript
• Lua
• Nim
• Perl
• PHP
• Python
• Ruby

std::ifstream ifs("path/to/some.gif", std::ifstream::binary);
kaitai::kstream ks(&ifs);
gif_t g = gif_t(&ks);

std::cout << "width = " << g.logical_screen()->image_width() << std::endl;
std::cout << "height = " << g.logical_screen()->image_height() << std::endl;

Gif g = Gif.FromFile("path/to/some.gif");

Console.WriteLine("width = " + g.LogicalScreen.ImageWidth);
Console.WriteLine("height = " + g.LogicalScreen.ImageHeight);

file, err := os.Open("path/to/some.gif")
g := NewGif()
err = g.Read(kaitai.NewStream(file), nil, g)

fmt.Printf("width = %d\n", g.LogicalScreen.ImageWidth)
fmt.Printf("height = %d\n", g.LogicalScreen.ImageHeight)

Gif g = Gif.fromFile("path/to/some.gif");

System.out.println("width = " + g.logicalScreen().imageWidth());
System.out.println("height = " + g.logicalScreen().imageHeight());

var g = new Gif(new KaitaiStream(someArrayBuffer));

console.log("width = " + g.logicalScreen.imageWidth);
console.log("height = " + g.logicalScreen.imageHeight);

local g = Gif:from_file("path/to/some.gif")

print("width = " .. g.logical_screen.image_width)
print("height = " .. g.logical_screen.image_height)

let g = Gif.fromFile("path/to/some.gif")

echo "width = " & $g.logicalScreen.imageWidth
echo "height = " & $g.logicalScreen.imageHeight

my $g = Gif->from_file("path/to/some.gif");

print("width = ", $g->logical_screen()->image_width(), "\n");
print("height = ", $g->logical_screen()->image_height(), "\n");

$g = Gif::fromFile("path/to/some.gif");

print("width = " . $g->logicalScreen()->imageWidth() . "\n");
print("height = " . $g->logicalScreen()->imageHeight() . "\n");

g = Gif.from_file("path/to/some.gif")

print("width = %d" % (g.logical_screen.image_width))
print("height = %d" % (g.logical_screen.image_height))

g = Gif.from_file("path/to/some.gif")

puts "width = #{g.logical_screen.image_width}"
puts "height = #{g.logical_screen.image_height}"

We maintain a growing free / open source repository of file formats and protocol specifications. Visit our format gallery to view the showcase of that repository with documentation, block diagrams and ready-made parser libraries in all supported target languages.

Format Gallery >>

Kaitai Struct is used in the following open source projects:

• Veles — binary data visualization and analysis tool
• mitmproxy — an interactive man-in-the-middle traffic inspection and modification tool
• Kismet — wireless network detector, sniffer, and intrusion detection system
• Beat Link Trigger — music performer app to trigger events based on Pioneer CDJs output
• Hobbits — multi-platform GUI for bit-based analysis, processing, and visualization
• OWASP ZAP — the world's most widely used web app scanner

If your project also uses Kaitai Struct, please drop us a line :)