File Modules (HEAD)

File Modules — More about file modules

Overview

File modules implement loading and saving of SPM files. The structure and operation of file modules will be explained on a simple but complete sample file module. This module imports a sample data format that was construed for this tutorial and does not occur in practice. However, the format resembles many (binary) data formats used in practice, except it contains no device and manufacturer information as they are not essential for the example.

The sample file format our module will load looks as follows, all values are stored in little-endian:

Furthermore we will assume files of this type get the extension .ssd (Simple SPM Data). The complete module source code follows, it will be described in detail below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152

#include <string.h>
#include <stdio.h>
#include <glib/gstdio.h>
#include <libgwyddion/gwymacros.h>
#include <libgwyddion/gwymath.h>
#include <libprocess/stats.h>
#include <libgwymodule/gwymodule-file.h>
#include <app/gwymoduleutils-file.h>

#include "err.h"

#define EXTENSION ".ssd"

#define MAGIC "SMPL"
#define MAGIC_SIZE (sizeof(MAGIC) - 1)

enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 };

typedef struct {
    guint xres;
    guint yres;
    gdouble measure;
    gdouble z0;
    gdouble gain;
} SimpleFile;

static gboolean      module_register(void);
static gint          simple_detect  (const GwyFileDetectInfo *fileinfo,
                                     gboolean only_name);
static GwyContainer* simple_load    (const gchar *filename,
                                     GwyRunType mode,
                                     GError **error);

static GwyModuleInfo module_info = {
    GWY_MODULE_ABI_VERSION,
    &module_register,
    N_("Imports simple data files."),
    "J. Random Hacker <hacker.jr@example.org>",
    "1.0",
    "Bit Rot Inc.",
    "2006",
};
GWY_MODULE_QUERY(module_info)

static gboolean
module_register(void)
{
    gwy_file_func_register("simple",
                           N_("Simple AFM files (.afm)"),
                           (GwyFileDetectFunc)&simple_detect,
                           (GwyFileLoadFunc)&simple_load,
                           NULL,
                           NULL);

    return TRUE;
}

static gint
simple_detect(const GwyFileDetectInfo *fileinfo,
              gboolean only_name)
{
    if (only_name)
        return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0;

    if (fileinfo->buffer_len > MAGIC_SIZE
        && memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0)
        return 100;

    return 0;
}

static GwyContainer*
simple_load(const gchar *filename,
            G_GNUC_UNUSED GwyRunType mode,
            GError **error)
{
    SimpleFile simple;
    GwyContainer *container = NULL;
    GwySIUnit *unit;
    GwyDataField *dfield;
    guchar *buffer;
    const guchar *p;
    GError *err = NULL;
    gsize size, expected_size;
    const guint16 *rawdata;
    gdouble *data;
    gint i;

    if (!gwy_file_get_contents(filename, &buffer, &size, &err)) {
        err_GET_FILE_CONTENTS(error, &err);
        g_clear_error(&err);
        return NULL;
    }

    if (size <= HEADER_SIZE) {
        err_TOO_SHORT(error);
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    if (memcmp(buffer, MAGIC, MAGIC_SIZE) != 0) {
        err_FILE_TYPE(error, "Simple");
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    p = buffer + MAGIC_SIZE;
    simple.xres = gwy_get_guint16_le(&p);
    simple.yres = gwy_get_guint16_le(&p);
    simple.measure = gwy_get_gfloat_le(&p);
    simple.z0 = gwy_get_gfloat_le(&p);
    simple.gain = gwy_get_gfloat_le(&p);

    expected_size = 2*simple.xres*simple.yres + HEADER_SIZE;
    if (size != expected_size) {
        err_SIZE_MISMATCH(error, expected_size, size);
        gwy_file_abandon_contents(buffer, size, NULL);
        return NULL;
    }

    simple.measure *= 1e-9;
    simple.z0 *= 1e-9;
    simple.gain *= 1e-9;
    dfield = gwy_data_field_new(simple.xres, simple.yres,
                                simple.xres*simple.measure,
                                simple.yres*simple.measure,
                                FALSE);
    gwy_convert_raw_data(p, simple.xres*simple.yres, 1,
                         GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN,
                         gwy_data_field_get_data(dfield),
                         simple.gain, simple.offset);

    unit = gwy_si_unit_new("m");
    gwy_data_field_set_si_unit_xy(dfield, unit);
    g_object_unref(unit);

    unit = gwy_si_unit_new("m");
    gwy_data_field_set_si_unit_z(dfield, unit);
    g_object_unref(unit);

    container = gwy_container_new();
    gwy_container_set_object_by_name(container, "/0/data", dfield);
    gwy_container_set_string_by_name(container, "/0/data/title",
                                     g_strdup("Topography"));
    g_object_unref(dfield);

    gwy_file_channel_import_log_add(container, 0, "simple", filename);

    gwy_file_abandon_contents(buffer, size, NULL);

    return container;
}

Administrative and Conventions

Beside standard headers our module includes a special header that provides some common inline functions

1

#include "err.h"

It is located in modules/file for the use by in-tree modules. External modules can copy it (or only the actually used functions) to their source code.

Header err.h contains common error reporting functions (called err_ERROR_TYPE). Its primary purpose it to ensure consistency of error messages between file modules and to ease translators' work.

Next we define a few convenience symbols: the file extension, the magic header and its size, and header size

1
2
3
4
5
6

#define EXTENSION ".ssd"

#define MAGIC "SMPL"
#define MAGIC_SIZE (sizeof(MAGIC) - 1)

enum { HEADER_SIZE = MAGIC_SIZE + 2*2 + 3*4 };

We also define a structure representing the imported file. It is a bit superfluous in our module as we do not pass this information around, but it is useful in many modules

1
2
3
4
5
6
7

typedef struct {
    guint xres;
    guint yres;
    gdouble measure;
    gdouble z0;
    gdouble gain;
} SimpleFile;

Registration

The feature registration is similar to data processing functions, there are just no menu paths and stock icons, on the other hand there are more functions the module can provide: one for each file operation. Our module implements only file detection and import therefore we pass NULL for the save and export functions.

1
2
3
4
5
6
7
8
9
10
11
12

static gboolean
module_register(void)
{
    gwy_file_func_register("simple",
                           N_("Simple AFM files (.afm)"),
                           (GwyFileDetectFunc)&simple_detect,
                           (GwyFileLoadFunc)&simple_load,
                           NULL,
                           NULL);

    return TRUE;
}

It is necessary to provide at least one of load, save and export operations, otherwise the file type would be rejected. It is highly recommended to provide a detection function, although it is not required. If no detection function exists for a file type the user has to always explicitly request the type to load and/or save files in this format.

Detection

Two types of detection exist: on load when we have the file available, and on save when the file does not exist and we know only the requested file name. They are differentiated by only_name argument. In both cases the detection function gets a GwyFileDetectInfo structure, but if only_name is FALSE, some fields are unset.

1
2
3
4
5
6
7
8
9
10
11
12
13

static gint
simple_detect(const GwyFileDetectInfo *fileinfo,
              gboolean only_name)
{
    if (only_name)
        return g_str_has_suffix(fileinfo->name_lowercase, EXTENSION) ? 20 : 0;

    if (fileinfo->buffer_len > MAGIC_SIZE
        && memcmp(fileinfo->head, MAGIC, MAGIC_SIZE) == 0)
        return 100;

    return 0;
}

The latter case is usually trivial, we just compare the extension with the normal extension of our file type.

The former case can be more complex. File name does not tell us much about file type: consider just the number of formats using extension .afm (not talking about .dat which can be essentially anything). Therefore we use file names only as the last resort and base the detection on actual file contents.

Many formats have well defined magic headers, precisely for the puropose of easy detection. If out format has one, we are lucky and the detection code can look as simple as in the example. To detect a particular version or type of a file format a more detailed analysis is necessary, but not substantially different from the simple case.

The most problematic case arises when the format has no detectable magic header at all. Often we can employ the following approach: read the header data and check whether it makes sense if interpreted as our file format. For example we read the number of lines and pixels per line, calculate expected file size and compare it to the real file size. Or we read the physical dimensions and check they are positive (and approximately of expected magnitude). If our detection routine requires significant processing or head and tail are not sufficient and we have to open the file ourselves, it is advisable to perform a quick check that can eliminate most files that are not of our type first, and then perform the sophisticated test only on the likely candidates.

Loading

The task of the load function is to read the file and create a GwyContainer containing all importable data. It must not assume anything about the file properties and contents, not even that the file passed the detection because the user can request a file type explicitly, bypassing detection. It has also deal with broken files, checking whether the file is long enough to contain the data it claims to and performing further parameter validation if necessary.

The file contents can be often read with gwy_file_get_contents() which uses mmap() (if available). The returned memory is read-only. Sometimes destructive header parsing is useful, in such a case it is possible to use g_memdup() to duplicate the header, keeping the data part mapped read-only.

1
2
3
4
5

if (size <= HEADER_SIZE) {
    err_TOO_SHORT(error);
    gwy_file_abandon_contents(buffer, size, NULL);
    return NULL;
}

The header is parsed using the portable binary buffer reading functions.

1
2
3
4
5
6

p = buffer + MAGIC_SIZE;
simple.xres = gwy_get_guint16_le(&p);
simple.yres = gwy_get_guint16_le(&p);
simple.measure = gwy_get_gfloat_le(&p);
simple.z0 = gwy_get_gfloat_le(&p);
simple.gain = gwy_get_gfloat_le(&p);

Then we can create the data field and read the data into it. Since Gwyddion always works with values in base SI units, we have to convert the raw data values to physical values. If the file uses non-base units (which is common in SPM), we have to multiply the data by the corresponding factor: in this case with 10^-9 as the physcial quantities are stored in nanometers in the file. Raw data conversion can be most easily performed with the convenience function gwy_convert_raw_data(). Sometimes it may be not sufficient so you can resort to the low-level GLib macros such as GUINT16_FROM_LE() or Gwyddion get-functions gwy_get_gdouble_be().

1
2
3
4
5
6
7
8
9
10
11

simple.measure *= 1e-9;
simple.z0 *= 1e-9;
simple.gain *= 1e-9;
dfield = gwy_data_field_new(simple.xres, simple.yres,
                            simple.xres*simple.measure,
                            simple.yres*simple.measure,
                            FALSE);
gwy_convert_raw_data(p, simple.xres*simple.yres, 1,
                     GWY_RAW_DATA_UINT16, GWY_BYTE_ORDER_LITTLE_ENDIAN,
                     gwy_data_field_get_data(dfield),
                     simple.gain, simple.offset);

We have to set also the lateral and value units of the created data field (by default, the dimensions and values are unitless).

1
2
3
4
5
6
7

unit = gwy_si_unit_new("m");
gwy_data_field_set_si_unit_xy(dfield, unit);
g_object_unref(unit);

unit = gwy_si_unit_new("m");
gwy_data_field_set_si_unit_z(dfield, unit);
g_object_unref(unit);

Our sample file format can only store topography channels, therefore the units are always meters. For different types of channels the units have to be set accordingly. Note that

1

unit = gwy_si_unit_new("nA");

is exactly the same as

1

unit = gwy_si_unit_new("A");

In other words, the prefixes are ignored because GwySIUnit represents the unit type and contains no magnitude information. However, if the format stores the units as a string including the prefix we can use gwy_si_unit_new_parse() to obtain the corresponding power of 10 (and subsequently multiply data field values with the correct multiplier).

1

unit = gwy_si_unit_new_parse("nA", &power10);

Finally we create a new GwyContainer, put the data field at "/0/data" and set the channel title "/0/data/title" (if the file contains explicit channel names, they are good candidates for a channel title, otherwise we settle for channel type as Topography or Current), and return the container. The application then takes care of adding it to the data browser and showing it in a data window. Calling gwy_file_channel_import_log_add() for channel 0 adds a data operation log entry, noting that the data originated by import from the "simple" format.

1
2
3
4
5
6
7
8
9
10
11
12

container = gwy_container_new();
gwy_container_set_object_by_name(container, "/0/data", dfield);
gwy_container_set_string_by_name(container, "/0/data/title",
                                 g_strdup("Topography"));

g_object_unref(dfield);

gwy_file_channel_import_log_add(container, 0, "simple", filename);

gwy_file_abandon_contents(buffer, size, NULL);

return container;

Loading More

Real file formats often contain more information we would like to import. First, they can contain several data fields (channels or even unrelated images). We store the second data at "/1/data" and its title at "/1/data/title", the third channel data at "/2/data", etc. The number sequence does not have to be contignuous but it is customary to make it so (in any case small integers should be used, do not start counting from 12758933).

Auxiliary information about the data as date and time of scanning, scan speed, tunneling current, tip oscillation frequency or user-defined comments can be stored to metadata. Metadata is a nested GwyContainer at "/0/meta" (for the first channel, the number of course corresponds to channel id). The keys are simply names of the values, the values are always strings. Note both keys and values have to be converted to UTF-8.

1
2
3
4
5
6
7

meta = gwy_container_new();
gwy_container_set_string_by_name(meta, "Comment",
                                 g_strdup(myfile->user_comment));
gwy_container_set_string_by_name(meta, "Tip oscillation frequency",
                                 g_strdup_printf("%g Hz", myfile->freq));
gwy_container_set_object_by_name(container, "/0/meta", meta);
g_object_unref(meta);

Portability

Gwyddion runs on many platforms and can be compiled with various compilers. File import is the area most affected by the differences between them, therefore certain care must be taken to make file modules work across all platforms.

Text Data

Text files can encode line ends differently from the platform Gwyddion is running on. Therefore opening files as text may lead to unexpected results. One usually opens all files as binary and uses gwy_str_next_line() to parse text files (or text parts of files) to lines.

The text representation of real numbers is locale-dependent. User's locale is unpredictable and a module must not set locale to arbitrary values because it would affect the rest of the application. Therefore one has to use functions as g_ascii_strtod() to read real numbers portably.

Many text strings, like comments and remarks, are stored in an arbitrary character encoding. Typically it is ISO 8895-1 (Latin1) or UTF-16, but sometimes more exotic encodings are used. Since Gwyddion uses UTF-8 for all text data (like Gtk+ does) they must be converted, e.g., with g_convert().

Binary Data

The canonical bad example is

1
2
3
4
5
6
7

struct {
    char c;
    LONG i;
    wchar_t remark[20];
} header;

fread(&header, sizeof(header), 1, filehandle);

For start, LONG is unlikely to exist on non-Microsoft platforms. So should we fix it to long int? No, the type size of classic C types (short, long, etc.) is different on different architectures (more precisely ABIs) and the standard prescribes only the lower limit. If we mean `32bit integer' we also have to express it in our code: gint32.

Likewise, wchar_t size is not well-defined. If the file comes from a Microsoft platform, where wchar_t is used a lot, it is probably 16bit, so we write gunichar2 instead and use g_utf16_to_utf8() to get UTF-8.

Then we have byte order. If we are on a platform with different native byte order than the file uses, we have to reverse the bytes in each multi-byte value. GLib provides macros such as GUINT16_FROM_LE() to simplify this task.

And last but not least, different ABIs prescribe different structure padding. Typically there can be from 0 to 7 unused bytes between c and i fields. While most compilers provide some packing #pragmas or other means to specify desired structure padding, the portability of these constructs is questionable at best.

The answer to (almost) all the problems above is: Don't do that then. Avoid direct reading of structures, read the data into a flat byte buffer instead and use the file module utils functions to obtain individual values.