Gwyddion – Free SPM (AFM, SNOM/NSOM, STM, MFM, …) data analysis software

Dump Format

Dump Format — Dumb dump file format used for data exchange between plug-in proxy and plug-ins

Purpose

The dumb dump file format is used for data exchange between plug-in proxy and plug-ins. It is not and never has been intended for permantent data storage. If you are looking for a simple file format Gwyddion can read and write then you are looking for GSF.

Overall Structure

The dump format consists of text lines, except for the actual data samples which are for efficiency reasons stored as arrays of (binary) IEEE double precision numbers.

Since dump files contain binary data (along with text lines) they have to be always opened in binary mode to avoid end-of-line conversions (on systems distinguishing between text and binary files). Unix end-of-lines are always used, i.e., lines are always terminated by a single LF (line feed) character, ASCII 0x0a. Plugin-proxy always writes data with Unix end-of-lines and although it may read dumps with CRLF end-of-lines without problems too, don't count on it.

String Data

The text lines have the form key=value. The keys are keys in data container created for the data, and the values are corresponding values. Except for a few special keys specified below, all values are stored as strings.

There are only a few types of data you may expect to find in or want to write to a dump file:

Data fields. They use some specific keys and you can read more about them below.
Metadata about the data. They start with /meta/ followed by an arbitrary key string. Not used directly by Gwyddion, but will appear in Metadata Browser.
Queer stuff (everything else). It's best to ignore it.

Data Fields

The data samples themselves are stored as a sequence of binary IEEE double precision floating point numbers in little-endian byte order. The array is preceded by line /0/data=[ and a single left bracket [ (with no newline after). The data samples are closed by a sequence of two right brackets ]] and a single LF character. The end of data field marker serves as a check the data was read correctly, as its size must be know beforehand.

To be more precise, /0/data is the name of the main data field. As data fields can generally have arbitrary names and left bracket is a valid value, a [ as a value marks start of a data field if and only if the next line starts with a [ too. Otherwise it is a regular metadata value.

The data samples are stored in `screen' order. That means they are ordered by lines (rows), starting from the uppermost one, and inside lines they are ordered by column, from left to right.

The resolutions and physical dimensions are specified using special keys /0/data/xres, /0/data/yres, /0/data/xreal, and /0/data/yreal. Since the resolutions must be obviously known before data samples can be read, the lines /0/data/xres=x-resolution and /0/data/yres=y-resolution must appear somewhere before them in the file. It is also strongly recommended to specify physical dimensions too, though it is not strictly necessary if you don't mind some bogus default value (like 1 meter) is substituted then.

All the values are in base SI units, i.e., dimensions are in meters (not micrometers, nanometres, or kilometres), currents in ampers (not picoampers), etc. The base units can be specified using the /0/data/unit-xy (lateral) and /0/data/unit-z (value, height) keys. If not specified, meters are assumed.

Example

A dump file with a single data field "/0/data" thus could look (replacing binary data with dots):

/0/data/xres=240
/0/data/yres=240
/0/data/xreal=4.1e-08
/0/data/yreal=4.1e-08
/0/data/unit-xy=m
/0/data/unit-z=m
/0/data=[
[...
...
...]]

This file would describe square data of size 240 × 240 samples and physical dimension 41nm × 41nm, with height as value.

If you want to pass mask (or presentation) instead of the main data, just replace the /0/data with /0/mask (or /0/show) in the previous description. If they are present in the data, they are always written to the dump, so you don't need anything special to read them.

Reference Implementations

A few sample dump format implementations are included in Gwyddion source distribution (see the directory plugins/file and the README there).

For several languages they were implemented as [language] modules that can be simply imported and used in your plug-in. The others have to carry the implementation with them if they want to use it.

Using library implementations directly

Following languages can benefit from language modules: Perl, Python, and Ruby. As of Gwyddion 1.6 their installation and import was finally standardized (in previous versions they were be installed into a hardly predictable directory that might or might not be in actual module search path for the language).

Plugin-proxy export an environment variable GWYPLUGINLIB that points to top-level directory of plug-in library modules. It contains a subdirectory for each language (perl, python, etc.). A plug-in should add corresponding subdirectory to its module search path – that is to @INC in Perl, sys.path in Python, and $: in Ruby. Then the module can be imported as usual. Please see the sample modules for possible implementations.

C

The plug-in proxy itself is always the most complete reference C implementation. See namely text_dump_export() and text_dump_import() functions in modules/plugin-proxy.c. Note this file has a special licensing exception, you can freely study it for the purpose of creation of Gwyddion plug-ins without becoming `tainted'.

C++

A sample C++ data-processing plug-in is included in Gwyddion distribution since version 1.4. It is divided to a simple class implementing dump file reading and writing (dump.cc, dump.hh) and the plug-in itself (invert_cpp.cc). These files are in the public domain.

Perl

A Perl module Gwyddion::dump handling reading and writing dump files is distributed as a part of Gwyddion since version 1.4. This module is in the public domain.

A sample Perl data-processing plug-in using Gwyddion::dump is included in the source distribution: plugins/process/invert_perl.pl. This script is in the public domain.

Python

A Python module Gwyddion.dump handling reading and writing dump files is distributed as a part of Gwyddion since version 1.4. This module is in the public domain.

A sample Python data-processing plug-in using Gwyddion.dump is included in the source distribution: plugins/process/invert_python.py. This script is in the public domain.

Ruby

A Ruby module gwyddion/dump handling reading and writing dump files is distributed as a part of Gwyddion since version 1.6. This module is in the public domain.

Two sample Ruby data-processing plug-ins using gwyddion/dump, one for vanilla Ruby and one making use of NArray, are included in the source distribution: plugins/process/invert_ruby.py and plugins/process/invert_narray.py. These scripts are in the public domain.

Pascal/Delphi

A sample Pascal/Delphi data-processing plug-in is included in Gwyddion distribution since version 1.4. The plug-in is implemented in the Delphi Pascal dialect, so it should work with FreePascal and Delphi.

It is divied to GwyddionDump unit (GwyddionDump.pas) implementing dump format reading and writing and the plug-in itself (invert_pascal.pas). These files are in the public domain.

FORTRAN

Work on a sample FORTRAN plug-in is in progress.