XVFS Build Walkthrough

xvfs-create

The script "xvfs-create" which is referenced below creates a C source file containing a local static structure and an Xvfs_<name>_Init() symbol. The C source file contains all the data from the specified directory, file names, contents, sizes, directory children. This C file can be compiled in a few different ways ("Client", "Standalone", and "Flexible") covered below.

See the final section for what this C source looks like.

Note the generated output will work for any mode (though it's unnecessary for "Server" mode, because "Server" mode does not supply any actual VFS data).

"Server"

Description

The "Server" mode is a single in-process handler for "//xvfs:/" and additional clients can call Xvfs_Register() to register their filesystem with that handler. This has the advantage of being more efficient since there is not one Tcl_Filesystem per extension, which have to all be searched through, but instead a single handler which uses a Tcl_HashTable to map the namespace to things which have called Xvfs_Register().

The "Server" mode thus has 2 symbols:

1. Xvfs_Init() which initializes the Tcl_Filesystem for all of "//xvfs:/" as well as setup the empty Tcl_HashTable
2. Xvfs_Register() which is for other extensions to call to register their name under "//xvfs:/"

Build Output

$ make xvfs.so
cc -I. -DUSE_TCL_STUBS=1 -DXVFS_DEBUG -I/usr/include  -DXVFS_MODE_SERVER -fPIC -g3 -ggdb3 -Wall  -o xvfs.o -c xvfs-core.c
cc -fPIC -g3 -ggdb3 -Wall   -shared -o xvfs.so xvfs.o -L/usr/lib64 -ltclstub8.6

$ ls -l xvfs.so
-rwxr-xr-x 1 rkeene users 14896 Sep 17 14:54 xvfs.so*

$ nm -D xvfs.so | grep ' [TU] '
0000000000004a07 T Xvfs_Init
0000000000004c6a T Xvfs_Register
                 U fprintf
                 U memcmp
                 U memcpy
                 U stderr
                 U strchr

Here we can see that indeed the only defined symbols for relocation are Xvfs_Init() and Xvfs_Register(), as well as some undefined symbols which we rely on. The symbols fprintf() and stderr are only used for debugging and would not be required in a production build.

"Client"

Description

The "Client" mode is very simple. It converts the specified directory ("example") into a C source file where all of the files contents and all directories children are structures of a local static structure. Then a Xvfs_example_Init() function is created which call Xvfs_Register() pointing to this structure (actually it points to some handlers so that the structure can change over time without relying on the core to be adaptable).

The "Client" mode thus exports a single symbol, Xvfs_<name>_Init(), and relies on a single symbol from the "Server" mode, Xvfs_Register().

Once you [load] this extension the Xvfs_example_Init() function is called and the VFS is registered under //xvfs:/example for all interpreters in this process.

Build Output

$ make example-client.so
./xvfs-create --directory example --name example > example.c.new
mv example.c.new example.c

cc -I. -DUSE_TCL_STUBS=1 -DXVFS_DEBUG -I/usr/include  -DXVFS_MODE_CLIENT -fPIC -g3 -ggdb3 -Wall  -o example-client.o -c example.c
cc -fPIC -g3 -ggdb3 -Wall   -shared -o example-client.so example-client.o -L/usr/lib64 -ltclstub8.6

$ ls -l example-client.so
-rwxr-xr-x 1 rkeene users 22584 Sep 17 14:54 example-client.so*

$ nm -D example-client.so | grep ' [TU] '
                 U Xvfs_Register
0000000000000ed5 T Xvfs_example_Init
                 U memcmp
                 U strlen

"Standalone"

Description

The "Standalone" mode is a logically combined "Client" and "Server" mode. However, it does not check for anything already handling "//xvfs:/" and always registers its own Tcl_Filesystem handling only //xvfs:/<name>. It does not export an Xvfs_Register() function and is completely standalone (hence the name).

It will work and can be [load]d on any version of Tcl, the same as "Client" mode but with no external dependencies on the "Server" mode.

The "Standalone" mode thus exports a single symbol, Xvfs_<name>_Init().

Build Output

$ make example-standalone.so
./xvfs-create --directory example --name example > example.c.new
mv example.c.new example.c

cc -I. -DUSE_TCL_STUBS=1 -DXVFS_DEBUG -I/usr/include  -DXVFS_MODE_STANDALONE -fPIC -g3 -ggdb3 -Wall  -o example-standalone.o -c example.c
cc -fPIC -g3 -ggdb3 -Wall   -shared -o example-standalone.so example-standalone.o -L/usr/lib64 -ltclstub8.6

$ ls -l example-standalone.so
-rwxr-xr-x 1 rkeene users 27680 Sep 17 14:54 example-standalone.so*

$ nm -D example-standalone.so | grep ' [TU] '
0000000000004d26 T Xvfs_example_Init
                 U fprintf
                 U memcmp
                 U memcpy
                 U stderr
                 U strlen

"Flexible"

Description

The "Flexible" mode is a combination of "Standalone" and "Client modes. It will search for an existing handler of "//xvfs:/" and if so call that handler's Xvfs_Register() function. If no such handler exists it will invoke the same handler as "Standalone" mode and register its own Tcl_Filesystem.

Because we cannot be linked with an undefined symbol to Xvfs_Register() the "Flexible" mode does not use the run-time linker to find the symbol for Xvfs_Register() but rather works with the Tcl VFS layer to locate that symbol dynamically.

The "Flexible" mode thus exports a single symbol, Xvfs_<name>_Init() and has no hard requirements on dependent symbols, however the "Server" mode's Xvfs_Register() will be called if its Xvfs_Init() function has been called previously to register a handler for //xvfs:/.

Build Output

$ make example-flexible.so
./xvfs-create --directory example --name example > example.c.new
mv example.c.new example.c

cc -I. -DUSE_TCL_STUBS=1 -DXVFS_DEBUG -I/usr/include  -DXVFS_MODE_FLEXIBLE -fPIC -g3 -ggdb3 -Wall  -o example-flexible.o -c example.c
cc -fPIC -g3 -ggdb3 -Wall   -shared -o example-flexible.so example-flexible.o -L/usr/lib64 -ltclstub8.6

$ ls -l example-flexible.so
-rwxr-xr-x 1 rkeene users 27728 Sep 17 14:54 example-flexible.so*

$ nm -D example-flexible.so | grep ' [TU] '
00000000000050c8 T Xvfs_example_Init
                 U fprintf
                 U memcmp
                 U memcpy
                 U stderr
                 U strlen

Generated Output

$ make example.c
./xvfs-create --directory example --name example > example.c.new
mv example.c.new example.c

$ ls -l example.c
-rw-r--r-- 1 rkeene users 47539 Sep 17 14:58 example.c

$ cat example.c
#include <xvfs-core.h>
#include <sys/stat.h>
#include <unistd.h>
#include <string.h>
#include <tcl.h>

#define XVFS_NAME_LOOKUP_ERROR (-1)
#define XVFS_FILE_BLOCKSIZE 1024

/*
 * XXX:TODO: Determine this automatically rather than
 *           by heuristics
 */
#define HAVE_STRUCT_STAT_ST_BLKSIZE 1
#define HAVE_STRUCT_STAT_ST_BLOCKS  1
#ifdef WIN32
#  undef HAVE_STRUCT_STAT_ST_BLKSIZE
#  undef HAVE_STRUCT_STAT_ST_BLOCKS
#endif

#define MIN(a, b) (((a) < (b)) ? (a) : (b))

typedef enum {
    XVFS_FILE_TYPE_REG,
    XVFS_FILE_TYPE_DIR
} xvfs_file_type_t;

typedef Tcl_WideInt xvfs_size_t;

struct xvfs_file_data {
    const char          *name;
    xvfs_file_type_t    type;
    xvfs_size_t         size;
    union {
            const unsigned char *fileContents;
            const char          **dirChildren;
    } data;
};

static struct xvfs_file_data xvfs_example_data[] = {
    {
            .name = "main.tcl",
            .type = XVFS_FILE_TYPE_REG,
            .size = 8730,
            .data.fileContents = (const unsigned char *) "\x23\x21\x20\x2f\x75\x73\x72\x2f\x62\x69"...
    },
    {
            .name = "foo",
            .type = XVFS_FILE_TYPE_REG,
            .size = 12,
            .data.fileContents = (const unsigned char *) "\x46\x6f\x6f\x20\x42\x61\x72\x20\x42\x61"...
    },
    {
            .name = "lib/hello/hello.tcl",
            .type = XVFS_FILE_TYPE_REG,
            .size = 77,
            .data.fileContents = (const unsigned char *) "\x6e\x61\x6d\x65\x73\x70\x61\x63\x65\x20"...
    },
    {
            .name = "lib/hello/pkgIndex.tcl",
            .type = XVFS_FILE_TYPE_REG,
            .size = 66,
            .data.fileContents = (const unsigned char *) "\x70\x61\x63\x6b\x61\x67\x65\x20\x69\x66"...
    },
    {
            .name = "lib/hello",
            .type = XVFS_FILE_TYPE_DIR,
            .size = 2,
            .data.dirChildren  = (const char *[]) {"hello.tcl", "pkgIndex.tcl"}
    },
    {
            .name = "lib",
            .type = XVFS_FILE_TYPE_DIR,
            .size = 1,
            .data.dirChildren  = (const char *[]) {"hello"}
    },
    {
            .name = "",
            .type = XVFS_FILE_TYPE_DIR,
            .size = 3,
            .data.dirChildren  = (const char *[]) {"main.tcl", "lib", "foo"}
    },
};

static long xvfs_example_nameToIndex(const char *path) {
    unsigned int pathHash;
    size_t pathLen;

    if (path == NULL) {
            return(XVFS_NAME_LOOKUP_ERROR);
    }

    pathLen = strlen(path);
    pathHash = Tcl_ZlibAdler32(0, (const unsigned char *) path, pathLen);
    switch (pathHash) {
            case 0:
                    if (pathLen == 0 && memcmp(path, xvfs_example_data[6].name, pathLen) == 0) {
                            return(6);
                    }
                    break;
            case 41419063:
                    if (pathLen == 3 && memcmp(path, xvfs_example_data[5].name, pathLen) == 0) {
                            return(5);
                    }
                    break;
            case 41877828:
                    if (pathLen == 3 && memcmp(path, xvfs_example_data[1].name, pathLen) == 0) {
                            return(1);
                    }
                    break;
            case 233898774:
                    if (pathLen == 8 && memcmp(path, xvfs_example_data[0].name, pathLen) == 0) {
                            return(0);
                    }
                    break;
            case 285410170:
                    if (pathLen == 9 && memcmp(path, xvfs_example_data[4].name, pathLen) == 0) {
                            return(4);
                    }
                    break;
            case 1197082414:
                    if (pathLen == 19 && memcmp(path, xvfs_example_data[2].name, pathLen) == 0) {
                            return(2);
                    }
                    break;
            case 1596983380:
                    if (pathLen == 22 && memcmp(path, xvfs_example_data[3].name, pathLen) == 0) {
                            return(3);
                    }
                    break;

    }

    return(XVFS_NAME_LOOKUP_ERROR);
}

static const char **xvfs_example_getChildren(const char *path, Tcl_WideInt *count) {
    struct xvfs_file_data *fileInfo;
    long inode;

    /*
     * Validate input parameters
     */
    if (count == NULL) {
            return(NULL);
    }

    /*
     * Get the inode from the lookup function
     */
    inode = xvfs_example_nameToIndex(path);
    if (inode == XVFS_NAME_LOOKUP_ERROR) {
            *count = XVFS_RV_ERR_ENOENT;
            return(NULL);
    }

    fileInfo = &xvfs_example_data[inode];

    /*
     * Ensure this is a directory
     */
    if (fileInfo->type != XVFS_FILE_TYPE_DIR) {
            *count = XVFS_RV_ERR_ENOTDIR;
            return(NULL);
    }

    *count = fileInfo->size;
    return(fileInfo->data.dirChildren);
}

static const unsigned char *xvfs_example_getData(const char *path, Tcl_WideInt start, Tcl_WideInt *length) {
    struct xvfs_file_data *fileInfo;
    Tcl_WideInt resultLength;
    long inode;

    /*
     * Validate input parameters
     */
    if (length == NULL) {
            return(NULL);
    }

    if (start < 0) {
            *length = XVFS_RV_ERR_EINVAL;
            return(NULL);
    }

    if (*length < 0) {
            *length = XVFS_RV_ERR_EINVAL;
            return(NULL);
    }

    /*
     * Get the inode from the lookup function
     */
    inode = xvfs_example_nameToIndex(path);
    if (inode == XVFS_NAME_LOOKUP_ERROR) {
            *length = XVFS_RV_ERR_ENOENT;
            return(NULL);
    }

    fileInfo = &xvfs_example_data[inode];

    /*
     * Ensure this is a file that can be read
     */
    if (fileInfo->type != XVFS_FILE_TYPE_REG) {
            *length = XVFS_RV_ERR_EISDIR;
            return(NULL);
    }

    /*
     * Validate the length
     */
    if (start > fileInfo->size) {
            *length = XVFS_RV_ERR_EFAULT;
            return(NULL);
    }

    if (*length == 0) {
            resultLength = fileInfo->size - start;
    } else {
            resultLength = MIN(fileInfo->size - start, *length);
    }
    *length = resultLength;

    /*
     * Return the data
     */
    return(fileInfo->data.fileContents + start);
}

static int xvfs_example_getStat(const char *path, Tcl_StatBuf *statBuf) {
    struct xvfs_file_data *fileInfo;
    long inode;

    /*
     * Validate input parameters
     */
    if (!statBuf) {
            return(XVFS_RV_ERR_EINVAL);
    }

    /*
     * Get the inode from the lookup function
     */
    inode = xvfs_example_nameToIndex(path);
    if (inode == XVFS_NAME_LOOKUP_ERROR) {
            return(XVFS_RV_ERR_ENOENT);
    }

    fileInfo = &xvfs_example_data[inode];

    statBuf->st_dev   = 0;
    statBuf->st_rdev  = 0;
    statBuf->st_ino   = inode;
    statBuf->st_uid   = -1;
    statBuf->st_gid   = -1;
    statBuf->st_atime = 0;
    statBuf->st_ctime = 0;
    statBuf->st_mtime = 0;
#ifdef HAVE_STRUCT_STAT_ST_BLKSIZE
    statBuf->st_blksize = XVFS_FILE_BLOCKSIZE;
#endif

    if (fileInfo->type == XVFS_FILE_TYPE_REG) {
            statBuf->st_mode   = 0100444;
            statBuf->st_nlink  = 1;
            statBuf->st_size   = fileInfo->size;
#ifdef HAVE_STRUCT_STAT_ST_BLOCKS
            statBuf->st_blocks = (fileInfo->size + statBuf->st_blksize - 1) / statBuf->st_blksize;
#endif
    } else if (fileInfo->type == XVFS_FILE_TYPE_DIR) {
            statBuf->st_mode   = 040555;
            statBuf->st_nlink  = fileInfo->size;
            statBuf->st_size   = fileInfo->size;
#ifdef HAVE_STRUCT_STAT_ST_BLOCKS
            statBuf->st_blocks = 1;
#endif
    }

    return(0);
}

static struct Xvfs_FSInfo xvfs_example_fsInfo = {
    .protocolVersion = XVFS_PROTOCOL_VERSION,
    .name            = "example",
    .getChildrenProc = xvfs_example_getChildren,
    .getDataProc     = xvfs_example_getData,
    .getStatProc     = xvfs_example_getStat
};

int Xvfs_example_Init(Tcl_Interp *interp) {
    int register_ret;

#ifdef USE_TCL_STUBS
    const char *tclInitStubs_ret;
    /* Initialize Stubs */
    tclInitStubs_ret = Tcl_InitStubs(interp, TCL_PATCH_LEVEL, 0);
    if (!tclInitStubs_ret) {
            return(TCL_ERROR);
    }
#endif

    register_ret = Xvfs_Register(interp, &xvfs_example_fsInfo);
    if (register_ret != TCL_OK) {
            return(register_ret);
    }

    return(TCL_OK);
}
#undef XVFS_NAME_LOOKUP_ERROR
#undef XVFS_FILE_BLOCKSIZE
$