#

FILENAME: binconvert.tcl

AUTHOR: erik.johnson@jogle.us

DESCRIPTION: binconvert is a package that reads & writes EEPROM memory files in multiple formats. It converts the data to & from a Tcl representation as a list of data segments, which is available for processing.

Currently supported formats are:

Intel hex format (I8, I16, & I32 subformats)
Motorola srec format (S19, S28, & S37 subformats)
TI txt format
raw binary format

This package documentation is auto-generated with Pycco: https://pycco-docs.github.io/pycco/

Use "pycco filename" to re-generate HTML documentation in ./docs .

package require Tcl 8.6
package require logger

#

binconvert

#

binconvert is a Tcl package that converts to & from a Tcl representation of a segment list. binconvert handles multiple binary formats and their subtypes. It currently handles:

"ihex" - Intel Hex format. Includes all three subtypes: I8hex, I16hex, & I32hex.
"srec" - Motorola SREC format. Includes all three subtypes: S19, S28, & S37.
"titxt" - TI .txt format, which has no subtypes.
"rawbin" - flat binary data, starting at address 0 on read, and also on write unless an execution start address is in the segment list.

Intel Hex format is documented on Wikipedia: https://en.wikipedia.org/wiki/Intel_HEX

Motorola SREC format is also documented on Wikipedia: https://en.wikipedia.org/wiki/SREC_(file_format)

Internal Data Representation

Binconvert stores its memory version of the binary data in a segment list. A Segment List is a list of paired elements. The even elements are segment addresses and the odd elements are blocks of segment data, represented as a list of numbers 0-255 that are the values of each byte of the segment. Being a list of bytes, endian issues are not relevant.

package provide binconvert $::BINCONVERT_PKG_VERSION

set dir [file dirname [info script]]

#

Usage

The package procedures are assembled into an ensemble command for external use. Normal usages are:

binconvert readfile format fileName (returns segmentList)
binconvert writefile format segmentList fileName
binconvert addparser fmtId lineParserName outputWriterName
binconvert loglevel newLogLevel

For Ihex and Srec files, writefile detects the address space needed by the Segment List and will automatically select the appropriate subformat for the actual data. In case the user wants to force their output into a particular subformat (e.g., write a file in I16hex format that could have fit in I8hex format), the individual subformats are also exposed, e.g.:

  binconvert writefile i8hex $segmentList $fileName
  binconvert writefile s27 $segmentList $fileName

Additionally, there is a hook for new user-supplied formats. The user must supply two procs: a line parser, and a output format writer. The interfaces are described below. Once those are defined, the user calls addparser as above, and the format called fmtId will be added to the list of available formats, and can be called in exactly the same way as the package-supplied formats. The perceptive reader will see that this interface is actually used to provision the package-supplied formats.

The loglevel command just exports the logger::setlevel command so the user can adjust the level of logging done by the package. For details and the list of valid levels, see the normal Tcl logger package documentation.

namespace eval ::binconvert {
    namespace export readfile
    namespace export writefile
    namespace export addparser
    namespace export loglevel
    namespace ensemble create

    logger::init ::binconvert
    logger::import -all -namespace ::binconvert::log ::binconvert
}

#

The list of valid formats and subformats supplied in the package is:

#

ihex
i8hex
i16hex
i32hex

source [file join $dir ihex.tcl]

#

srec
s19
s28
s37

source [file join $dir srec.tcl]

#

titxt

source [file join $dir titxt.tcl]

#

rawbin

source [file join $dir rawbin.tcl]

#

loglevel

#

A simple accessor to change the logging level of the package for debugging.

proc ::binconvert::loglevel {newLevel} {
    ::binconvert::log::setlevel $newLevel
}

#

The default log setting for binconvert is warning or higher.

::binconvert::loglevel warn

#

readfile

#

readfile accepts as its arguments the name of an input file and a token designating which format to parse it as. It will attempt to open the file, read the contents, and parse them into a segment list. A new segment is begun with each new segment offset/address command in the input, or when there is a discontinuity in the sequential data addressing even if the segment address was not changed by an address command.

proc ::binconvert::readfile {datafmt fname} {
    set resultL [list]
    set segData [list]
    set ::binconvert::IhexFileType ""
    set ::binconvert::SrecLineCount 0
    set address 0
    set segAddr 0
    set startAddress 0

#

Select a parsing format

Based on the data format argument, select the correct line parser. The formats are added using addparser, as can be seen below. Package supplied line parsers are:

Intel Hex ("ihex"): ProcessIhexLine
Motorola SREC ("srec"): ProcessSrecLine
TI txt ("titxt"): ProcessTitxtLine
Raw Binary ("rawbin"): ProcessRawbinInput

    if {![info exists ::binconvert::Formats($datafmt,parser)]} {
        log::error "binconvert::readfile: Unknown data format $datafmt."
        error "Unknown data format $datafmt"
    }
    set parser $::binconvert::Formats($datafmt,parser)

#

Read input file

The input file is opened read, and split into a list of lines. Because these formats are normally only used on relatively small files, we're temporarily storing the whole thing in memory. If you run out of memory here, your file is on the order of a Gigabyte or more... and these are really not appropriate formats for something that large.

    set cnt 1
    set chan [open $fname r]
    try {

#

Unless the data is raw binary, split the input data into lines. If the format is raw binary, there's no lines to split, but we need to read it as binary.

        if {$datafmt ne "rawbin"} {
            set dataS [read $chan]
            set dataL [split $dataS \n]
            log::debug "File read.  [llength $dataL] lines found."
        } else {
            chan configure $chan -translation {binary binary} -encoding binary
            set dataS [read $chan]
        }
    } finally {
        chan close $chan
    }

#

Parse input data

Note that actual input parsing here is done by the parser selected above. Each parser accepts one line of data and emits a list that consists of a canonical tag and its associated data.

The specification for the interfaces to the procedures are as follows: the line parser must take exactly one argument, the line to be parsed. It must return a list consisting of one data type token plus one or two data fields appropriate to the token. The list of supported tokens is:

DATA lineAddress dataList
SEGADDR segAddressOffset
HEADER headerData
NOP {}
STARTADDR executionStartAddress optionalProcessorType
EOF {}

The readfile level manages the segment list structure based on those tags. Format-specific consistency checks are handled in the parsers.

    foreach line $dataL {
        set lineL [{*}$parser $line]
        switch -- [lindex $lineL 0] {
            default {

#

Unknown line types are warned about, but we continue and allow the user to decide what to do about the issue.

                log::warn "binconvert::readfile: Unknown line type\
                            [lindex $lineL 0] in line: $line"

#

Data line.

Check the address. If it's not continuous with the current segment, wrap up the current segment and init a new segment. Either way, after any fix-up work, append the line data to the current active segment.

            }
            DATA {
                log::debug "DATA: $lineL"
                if {([lindex $lineL 1] + $segAddr) != $address} {
                    if {[llength $segData] != 0} {
                        lappend resultL $startAddress $segData
                    }
                    set address [expr {[lindex $lineL 1] + $segAddr}]
                    set startAddress $address
                    set segData [list]
                }
                lappend segData {*}[lindex $lineL 2]
                incr address [llength [lindex $lineL 2] ]

#

Segment Address line.

New segment offset address, where segAddr is an offset that will be added to all line addresses until the next segment addr line.

Wrap up any ongoing segment and init a new segment.

            }
            SEGADDR {
                log::debug "Found SEGADDR [lindex $lineL 1]"
                set segAddr [lindex $lineL 1]
                if {[llength $segData] != 0} {
                    lappend resultL $startAddress $segData
                }
                set address $segAddr
                set startAddress $address
                set segData [list]

#

Start Address line.

Contains processor address from which to start execution, if this was a processor.

Note: All start address types map to this. Because it's relevant to their output format, I16hex and I32hex include an extra field in their output with the format type. This is passed along and must be dealt with in the writers.

N.B. that the "address" for this segment is the text string "STARTADDR", not an actual numeric address, and the data is a single execution address rather than a byte list.

            }
            STARTADDR {
                log::debug "Found STARTADDR [lrange $lineL 1 end]"
                if {[llength $segData] != 0} {
                    lappend resultL $startAddress $segData
                }
                set address "STARTADDR"
                set startAddress $address
                set segData [lrange $lineL 1 end]

#

Header line.

Srec header line. Data is still a list of hex bytes, but usually holds a null terminated string. The data is saved in the segment list, but will be output only if the data is written in Srec format.

N.B. that the "address" for this segment is the text string "HEADER", not an actual numeric address.

Note also that the header is specified as the first line of a strictly formatted SREC file. If other lines precede it, we will emit a warning and discard any prior data, but we will continue processing.

            }
            HEADER {
                log::debug "Found HEADER [lindex $lineL 1]"
                if {[llength $segData] != 0} {
                    log::warn "binconvert::readfile: Data in file before header: line $cnt."
                    log::info "binconvert::readfile: Data was $segData"
                    set segData [list]
                }
                set address "HEADER"
                set startAddress $address
                lappend segData {*}[lindex $lineL 1]

#

EOF line.

Force end of processing. No more lines (if any) will be processed.

            }
            EOF {
                log::debug "Found EOF"
                break

#

No-op / No segment data

Certain lines exist for format internal reasons (e.g., SREC count records) and have no data that goes into the Segment List.

#

Put final segment on result list and return it.

    if {[llength $segData] != 0} {
        lappend resultL $startAddress $segData
    }
    return $resultL
}

#

writefile

#

writefile will examine the file format token, then call the appropriate output format writer for that file type. Prepend any special writer arguments to any extra caller args when calling the writer.

proc ::binconvert::writefile {datafmt segmentList fileName args} {
    if {![info exists ::binconvert::Formats($datafmt,writer)]} {
        log::error "binconvert::writefile: Unknown data format $datafmt."
        error "Unknown data format $datafmt"
    }
    set outchan [open $fileName w]
    try {
        {*}$::binconvert::Formats($datafmt,writer) $segmentList $outchan \
                    {*}$::binconvert::Formats($datafmt,writeArgs)  {*}$args
    } finally {
        chan close $outchan
    }
}

#

addparser

#

addparser accepts three arguments:

a format symbol, used to identify when this format is desired,
the name of a line parsing procedure
the name of an output writer procedure

It takes the procedure names and stores them so that they can be called whenever readfile or writefile is called with that format symbol. See just below for a usage example.

The specification for the interfaces to the procedures are as follows: the line parser must take exactly one argument, the line to be parsed. It must return a list consisting of one data type token plus one or two data fields appropriate to the token. The list of supported tokens is:

DATA lineAddress dataList
SEGADDR address
HEADER headerData
NOP {}
STARTADDR executionStartAddress optionalProcessorType
EOF {}

Each data line is expected to have an address. Several formats support some idea of a segment address that is then added to all data lines until a new segment address line is seen. Some formats have a record for the address to load into the processor to start execution; these vary but are combined into a single conceptual thing. It can be tagged with a format identifier; due to differences between processors, changing format to one that does not normally support a tagged execution start address may be hazardous; the user must use their judgement in this case.

N.B. that these are all in the shared ::binconvert:: namespace. When a user adds a new format, they must either define their procedures in the ::binconvert:: namespace, or must provide a fully-qualified namespace path to their procedures.

proc ::binconvert::addparser {formatSymbol parseProcName writeProcName args} {
    set ::binconvert::Formats($formatSymbol,parser) $parseProcName
    set ::binconvert::Formats($formatSymbol,writer) $writeProcName
    set ::binconvert::Formats($formatSymbol,writeArgs) $args
}

#

Here is where we actually set up all the package-supplied formats.

::binconvert::addparser ihex ProcessIhexLine WriteIhexFile
::binconvert::addparser i8hex ProcessIhexLine WriteI8hexFile
::binconvert::addparser i16hex ProcessIhexLine WriteI16hexFile
::binconvert::addparser i32hex ProcessIhexLine WriteI32hexFile
::binconvert::addparser srec ProcessSrecLine WriteSrecFile
::binconvert::addparser S19 ProcessSrecLine WriteSrecFile S19
::binconvert::addparser S28 ProcessSrecLine WriteSrecFile S28
::binconvert::addparser S37 ProcessSrecLine WriteSrecFile S37
::binconvert::addparser titxt ProcessTitxtLine WriteTitxtFile
::binconvert::addparser rawbin ProcessRawbinLine WriteRawbinFile

unset ::BINCONVERT_PKG_VERSION