FILENAME: autoObject.tcl

AUTHOR: theerik@github

DESCRIPTION: AutoObject is a TclOO class-based package used to create auto-assembling objects that map a Tcl dict-style get/set interface onto a block of unstructured memory using simple run-time descriptors. The key feature is that objects can be transparently serialized and deserialized to/from binary or byte list format, allowing them to be passed to or received from any interface that supports byte list/serial formats, while still being Tcl-intuitive to work with programmatically.

          Past applications include converting structured data to/from
          byte arrays for COM interfaces to other languages, message
          formats for serial communication, and parsing of memory
          blocks retrieved from embedded targets.

Copyright 2015-19, Erik N. Johnson

This software is distributed under the MIT 3-clause license.

This package documentation is auto-generated with Pycco: https://pycco-docs.github.io/pycco/

Use "pycco filename" to re-generate HTML documentation in ./docs .

autoObject base class

Creates objects that use a defining list to: set up field keys that reference fields, mixin a class to support the type of data in that field, delegate field access to the methods of the mixin class, and initialize the fields to their default values.

The field object types must support canonical methods to determine how to parse or generate serialized format, to programmatically set or get field values, and to output human-readable values for logging and debug.

While some objects are created with only the base autoObject class, most uses/users will extend the base class for the particular needs of their application; e.g. message or data cache or COM object interface or....

The base class provides the following methods, which may be overridden or extended by subclasses but are expected to be callable:

• $object get ?fieldName?
• $object set fieldName ?value? ?fieldName value?
• $object toString
• $object fromList byteList
• $object toList
• $object fromBinary binaryString

• $object toBinary

• $object follow

• $object createWidget

• $object configure option value ?option value* ...?

• $object cget ?option?

All autoObject objects support the Tk-style configure/cget interface for options. The base class options are only -initialized, which indicates if this object has had any fields set since creation, and -level, which sets the level of log messages (per the logger package, https://core.tcl-lang.org/tcllib/artifact/06eec89ccb71c91c). Other options can be added by subclasses, of course.

The following methods are mostly used by the constructor, but are exposed to allow programmatic use. Do not use unless you know what you're doing.

• $object addField fieldName fieldDefinitionList
• $object deleteField fieldName
• $object validate

The following methods are used for internal purposes only:

*

autoObject constructor

The constructor takes as input a list defining the object to be created. The defining list has the form of key/valueList pairs, with field names as keys paired with a field definition list as value. The field definition list is a list of up to 6 elements, the first 3 of which are mandatory:

{ field_offset  # Must start at 0, incr by field_size,
                # leave no gaps.
  field_size    # (Max) size of field in bytes in serial
                # input format.
  field_type    # Name of the data type. Custom types
                # must support base methods used by
                # autoObject types, may have others.
  init_value    # Value used to initialize field data.
                # May be empty {}.
  field_data    # Passed to constructor when creating the
                # object of type <field_type>, not used or
                # examined by autoObject.  May be empty {}.
  widget_name   # Name of an autoWidget class to use to
                # display the data in GUIs.  May be empty,
                # in which case the autoWidget "autoEntry"
                # will be used if needed.
}

If fewer than 6 elements are provided, the remainder will be filled with default values as noted, usually the empty list {}.

There is one special reserved field name, "variable_length_object", whose value is a single number defining the minimum size of the object. If the value == 0, it is treated as disabling the special handling (i.e., the object cannot be variable-size); otherwise, too-small input is changed to a minimum of $value, and fields past that size are permitted to be variable in size. I.e., the object will accept a serial input of any size between $value and blockSize without complaining, filling fields in order until it runs out of input. If the last field is given some data but not enough bytes to fill it correctly, it is left to the field object to complain; the containing autoObject will not.

There are multiple times we'll want the list of field names sorted by the field offset, so we do the sort now and save it for the future.

Iterate over the sorted field list and add each field to the object. Keep a running list of the names in sorted order and the total of the size in bytes of the serialized data block. If we see it, process the special token "variable_length_object" separately - we don't want to treat it like a normal field.

After we read the (sorted) field list to initialize the fields, parse the sizes and offsets to validate the input. Important: there cannot be any missing bytes in the structure.

autoObject Field Engine

In the first two generations of design, field construction was the core activity of the object constructor. However, to allow subclassing or object refinement operations to use the main autoObject engine to dynamically add new fields to an existing object, the addField engine is now separately exposed.

Warning: This interface was originally designed to only be called by the constructor, and it should NOT be called lightly. Adding fields without knowing what you're doing will almost certainly break your objects. In particular, normal object validation is not done, and the NameL field that is used to drive the to/from/List/String/etc. methods has to be correctly modified to come back into sync with the field data, so that every byte address is in exactly one field and the list is correctly sorted by starting offset. That said, if you need to do it, then this is how.

Most commonly, a type that expects to be subclassed later will place the uninterpreted bytes in a byte array field called something like "payload", which would need to be removed from NameL (using deleteField) and replaced by all the fields that comprise that data when subtyped. Usually the subtype would determined by either a code defined in the container, or by some field inside the contents of "payload", but I'm sure this will be false in some cases I haven't thought of yet.

Note that adding fields only sets data to default values, and deleting fields deletes their data. Also note that the object must be manually validated to ensure stuctural integrity. If the object is meant to be mutated in place, retaining its data, the user should cache the serial data separately during the mutation, performing a sequence generally like the following example:

proc mutateObject {objName} {
  uplevel $objName obj
  set dataL [$obj toList]
  $obj deleteField *fieldName*
  $obj addField *fieldName1* *fieldDefiningList1*
      ...
  $obj addField *fieldName2* *fieldDefiningList2*
  $obj validate
  $obj fromList $dataL
}

autoObject.addField

addField takes a field name and its defining list as inputs. It defines all of the FieldInfo values that allow the data operations to be carried out, and initializes the data to its init value, if any. It does not validate the object afterwards; this is done by the constructor or must be done manually by the user.

Find the data type corresponding to the name. Start by parsing out any array syntax. Then look in various namespaces to find a definition of the type.

1) Is the type declared in the appropriate namespace? If so, use it.

2) Is the type defined in a different namespace? If so, we either try it or die in error, so we may as well keep going and try it.

3) Can't find a mixin class with that name. Error out.

If it's not mixed in already, append the new type mixin class

If the field type defines an InitField method, run it using the typeData field in the defining list

Forward all field references to the Dispatcher, which requires the field name and mixin type namespace. Different methods are used for scalar fields and arrays.

Initialize the new field to its default value. First, check to see if field is a scalar value or an array. If it's scalar, just init it.

It's an array. Validate that the total specified data size is an integral number of bytes per entry.

If we weren't given an exact list of initializer data, one entry per element, assume what we were given is for one element. Replicate it to cover all elements.

Now look for the defined widget, if any.

Is it "self"? If so, easy.

Is the widget declared in the appropriate namespace? If so, use it.

Found something not in the right namespace; we either try it or die in error, so we may as well keep going and try it.

Can't find a widget with that name. Error out.

Widget wasn't defined; use autoEntry as the base alternative.

Mix in the UI class if {$widgetName ne "self"} { oo::objdefine [self] [list mixin $widgetName] } Final cleanup

autoObject.deleteField

deleteField takes a field name and deletes all the data and parsing information about that field. This could allow other fields to be added to replace the deleted field, but that's a higher-level concept. N.B. that the data for the removed field is destroyed, and must be saved at the higher level if the object is being mutated in place.

autoObject.validate

validate attempts to check the object for consistency, making sure that no byte is assigned to multiple fields or to no field, all fields are in NameL, and that the total blocksize matches the field definition.

Check to see that all fields are defined in FieldInfo, and that no extra fields are hanging around.

Iterate across NameL and check: start_address == last end_address end_address = start_address + size * last end_address == defined blockSize

autoObject Base Operations

All autoObjects support a base set of subcommands: get, set, toList, fromList, and toString. Special handling for each of these is delegated to the class of the field data type, with the common structure implemented in the base class.

autoObject.get

Field getter: accepts an args list of keys, and returns a list of values for each key provided or a single value for a single key. If no key is provided, returns the entire data array in a single list with alternating keys & values, as if from array get. Warns on unknown keys or uninitialized object.

No keys: return name/value pairs for all defined fields.

Single key: look up field, return value

autoObject.set key/valueList

Simple mutator: accepts a list of key/value pairs, and for each key sets the associated value to the supplied value. Warns on but does not reject unknown keys. Unknown keys have values stored in the data array and can be retrieved with get, but are not involved in toList/fromList operations.

autoObject.toString

Getter to screen format. For each field, retrieves the value and appends it to the output string in the style defined by the type in the field array. Displays fields not defined in the defining array at the end, marked as special.

This can be overridden by derived classes to create a different default format, but has a nominal implementation in place: name: value, where the colon is 36 columns from the left margin.

Iterate over the ordered list of field names created by the constructor.

Be sure to document any extra fields in the data dict

autoObject.toList

Getter to byte-list format (a list of numbers 0-255 representing bytes). Gets the byte-list representations of the field objects in the order defined by the defining array, collects them into a combined byte list, and returns the list.

Iterate over the ordered list of field names created by the constructor.

List should have been the right size coming into each field.

set dataL {} foreach obj $DataArray($name) { lappend dataL {*}[$obj toList] }

Field object should generate correct list length.

Just in case, convert to list of non-negative byte values (range 0-255)

autoObject.fromList inputList

Setter from byte-list format, as supplied by a decoded message or memory dump. Accepts the byte list, parses it as defined in the associated defining array, and stores it in the object's data array.

Check that input is correct size for object

In all cases, too large is an error

Block is too small and object is not variable-length

Variable size is allowed to be smaller but not larger

Special case - clear out the existing data, because we won't loop past the end of the 0 byte input block.

Input is acceptable. Process it in field order.

Once we're past the end of the list, stop processing.

Cut off the next field's data from the input list

Single value field

autoObject.toBinary

To e.g. wire format or COM interface. Calls toList to collect the object field data, then converts from byte-list format to binary string.

autoObject.fromBinary

From e.g. wire format or COM interface. Converts from binary string to byte-list format, then calls fromList to populate the object fields.

autoObject.configure

Support for the traditional Tk-style configure operator, which can change the values of the various object options. Pairs with cget.

The base class supports the following options:

• -initialized: sets the internal Initialized flag. Initialized is set false on creation, set true on fromList, fromBinary, or set. This only needs to be overridden to allow a object with all fields still at default values to be serialized.
• -level: sets the level of logging, per the logger package. Possible values are : debug, info, notice, warn, error, critical, alert, and emergency. Nothing above error is used in this package. For details, see https://core.tcl-lang.org/tcllib/artifact/06eec89ccb71c91c).

To extend this in a subclass, I recommend beginning by calling next to allow the base case to handle the errors and its own options, then (assuming no errors) handling your own options. E.g.:

  method configure {args} {
      next {*}$args
      foreach {opt value} $args {
          switch -exact -- $opt {
              -myOption {
                  # does things
              }
          }
      }
  }

Also, note that the Options base class variable should have your new options appended in the constructor. E.g.:

  my variable Options
  lappend Options "-myOption"

autoObject.cget

Support for the traditional Tk-style cget operator, which returns the values of the various object options. Pairs with configure.

The base class supports the following options:

• -initialized: returns the internal Initialized flag. Initialized is set false on creation, set true on fromList, fromBinary, or set.
• -level: returns the level of logging, per the logger package. Possible values are : debug, info, notice, warn, error, critical, alert, and emergency. For details, see https://core.tcl-lang.org/tcllib/artifact/06eec89ccb71c91c).

To extend this in a subclass, I recommend beginning by checking your own options first, then falling through on default to call next to allow the base case to handle the errors and its own options. Also, the Options class variable should have your new options appended in the constructor as noted above. E.g.:

  method cget {option} {
      switch -exact -- $option {
          -myOption {
              return $MyOptionValue
          }
          default {
              return [next $option]
          }
      }
  }

autoObject.follow

Returns a fully qualified path to the value of the data object; normally used by GUIs to use a field as a textvariable.

autoObject.exists

Modelled after dict exists, returns true if the key exists in the object's data definitions.

autoObject.map

Modelled after dict map, applies a transformation script to each element of the array, returning the list of transformed values in order.

autoObject.size

Modelled after dict size, returns the number of key/value mappings in the object as currently defined.

autoObject Dispatcher Engine

Note: this section is subject to change. There still feels like there's a major simplification or optimization left to find that will collapse the dispatch process. But until I figure out the right trick, this solution is proven to work.

When a client invokes the field-specific operator syntax: object fieldName operator value the object must determine if the dataType of fieldName has a method that matches operator, and if so it must invoke it and pass along all the other arguments.

To accomplish this, when we built the object we forwarded the method fieldName to either Dispatch (for fields with scalar values) or to ArrayDispatch (for fields which consist of a list/array of values).

autoObject.Dispatch

When the object forward that supports scalar Dispatch was created, the object method "fieldName" was mapped to "Dispatch fieldName fieldType" in order to provide the actual type of the field directly, rather than requiring Dispatch to dereference the type in the FieldInfo dict. This allows scalar Dispatch to run in effectively one command.

autoObject.ArrayDispatch

Array dispatch is more complex than scalar dispatch because even the basic operations become complex when they must be applied to all members of a list of data. Depending whether this is a data access, a data modification, or a read-modify-write, the logic is different.

Therefore, every command that supports arrays must have a special Dispatch helper. These procs are defined in the AutoObject::Dispatch namespace to prevent collision with common operators of the same name. When ArrayDispatch is called, with the same operands in the same order as scalar Dispatch, it calls the proc in the AutoObject::Dispatch namespace whose name is formed by prepending the string "Array" to the string naming the operator, e.g. "Arrayset" for "set", passing along all other arguments including fieldName and dataType.

The base AutObject package provides operator procs for all the field operators supported by the scalar fields (set, get, toList, fromList, toString), plus additional array-only operators supporting list operations (lset & lindex; lappend was considered but did not make sense in the context of a fixed-length structure). Type-specific operators can be added but must be defined in the correct namespace to be located by ArrayDispatch.

autoObject.Dispatch.Arrayset

Set is one of the more complex operations in an array. To keep the model manageable, we insist on a simplification: if the user wishes to set the value of an array, which by the data definition is a list of predefined length, they must provide exactly that many elements as input to the command. If the length is incorrect it is considered an error.

Note that to support data types that have limitations on the data that can be represented, every data type must support the ValueFilter method, which applies the relevant transform to the input data that allows it to be represented in the data. (E.g., a uint8_t data type would set a value to its value modulo 256, reflecting how it would be represented in the serial memory space governed by the object.) This ValueFilter method is applied to each data element in the input list to create the final field value.

autoObject.Dispatch.Arrayget

By contrast to "set", "get" is the conceptually simplest array operation. It returns a copy of the full list containing the value of the field.

autoObject.Dispatch.ArraytoString

The toString operator converts each element of the array to the nominal string representation for its data type. Because the data methods only operate on a single element, the data type's toString operator must be accumulated with an lmap. The string elements are joined with spaces to form the final string representation.

autoObject.Dispatch.ArrayfromList

The fromList operator has to distribute the data across all the elements of the array. It does this by using the convention that every dataType must accept a data list as the argument for fromList, then return the remaining list after having removed exactly the data necessary. This remaining list can be then passed as input to the next element, etc., until all elements are populated. Any remaining data is returned per the same convention.

autoObject.Dispatch.ArraytoList

The toList operator constructs its output by iterating across the elements of the array with an lmap and appending the list representation of each element in turn, as defined by its passed-in data type. Since many data types return lists, which are appended entire, we flatten the final result to a single list of bytes.

autoObject.Dispatch.Arraylset

The lset method works like the native command or the dict command: it sets the list element at the provided index. Since the last element is the new value, that element is sent through the ValueFilter to ensure that the value can be represented in the memory block.

autoObject.Dispatch.Arraylindex

The lindex method works like the native command: it returns the n'th element of the array. While the lindex command can nest indices, this makes no sense in the context of the fixed-length array, and is not supported.

autoObject.createWidget

Returns a Tk name of a frame that encloses a grid of name/value paired widgets. Names are labels; values are whatever widget the field object has mixed in that responds to the createWidget method call. Individual field widgets have the option to add a widget to a special list of large widgets that show up at the bottom of the grid (BigWidgetL).

Create encapsulating frame

@@@ TODO %%% llength NameL is not really the right number for number of rows - doesn't take into account dropped reserved fields or arrays/bitfields that take multiple rows. Figure out a better count later. Cap it at 20 to ensure it fits on most monitors and distribute evenly across columns.

Iterate over all fields, creating pairs of labels (for name) and display widgets (for values).

First, handle non-array fields

Special support for reserved fields - if no widget is returned, don't try to grid it or add rows.

If creation returns 2 widgets, the first is for the small in-row pace, the second is the large breakout version

Handle field arrays by creating one name label for all, and one value display widget per object in the array.

Special support for reserved fields - if no widget is returned, don't try to grid it or add rows.

If creation returns 2 widgets, the first is for the small in-row pace, the second is the large breakout version

Columns over 20 items get unwieldy, and don't fit well on smaller monitors. @@@ TODO %%% make this value configurable & save in .ini file.