Marpa

Timeline
Login
Tcl 2016 Conference, Houston/TX, US, Nov 14-18
Send your abstracts to tclconference@googlegroups.com by Sep 12.

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

50 most recent timeline items

2018-04-20
21:28
[faebf8fba4] Leaf: inbound: Absolute movement, tweaked for consistency. Extended with optional delta arguments. gate: Extended with forwards to the input location accessor and modifiers. This provides parse event handlers with the ability to move in the input. lexer: Moved redo to before we handle parse events. This enables parse event handlers to modify the location without interference from the system itself. Modified pre-lexeme event generation to move input to the start of the lexeme. Fixed pe-fill method, forgot to set flag for the incremental rebuild of the symbol/sv tables. parse descriptor: See gate. Further fixed accessor setup. Extended view to report input location. testsuite: Report input location. Move input location to the end of the lexeme. Needed now that pre-lexeme events have the location at the start. Updated expected results. (user: aku, tags: parse-events)
04:33
[07ca2e2be0] Pulled memory fixes into the feature branch implementing parse events. (user: aku, tags: parse-events)
2018-04-19
19:49 • Edit [cc42a785875a7adf|cc42a78587]: Edit check-in comment. (user: aku)
19:47
[dcf340cd32] Leaf: Update main line with the accumulated memory fixes. (user: aku, tags: trunk)
19:46 • Edit [a29c4613b0ad2de3|a29c4613b0]: Mark "Closed". (user: aku)
19:44
[dc34e1afbb] Closed-Leaf: Merged series of fixes for memory issues (memory leaks, management mismatches, ...). Together with the push of memory-intensive tests into child processes done here the testsuite should now be properly constrained again wrt memory usage. Updated tests. (user: aku, tags: mem-limit)
18:18
[a29c4613b0] Closed-Leaf: Added test against unbound parser memory usage in parser-core. Found and fixed SV ref-count mismanagement (RCM) in the lex-only code path of the RTC. Added narrative tracing to the code path. Cleanup of lexer-core testsuite with regard to memory debugging. Found and fixed Tcl_Obj* RCM in the lex-only critcl template. Added narrative tracing to the template. (user: aku, tags: memory-cleanup)
2018-04-18
23:41
[714d438603] Do not mix malloc and Tcl allocation routines. Even when the malloc is hidden, here in `strdup`. Used to allocate strings with strdup, then release with ckfree/Tcl_Free. This messed up memory mgmt internals to the point of memory smashes. Fixed, now using a our own implementation of strdup based on the allocation macros from environment.h. This ensures that alloc and free match. Thank you, valgrind. (user: aku, tags: memory-cleanup)
03:08
[ddc1e67640] Moved the main parts of `test/support/gen.tcl` into `bin/i-gen`. This new internal generator application uses the public `marpa-gen` as the underlying workhorse. The support code now invokes the internal generator instead of doing everything itself. With the operation confined to a child process the amount of memory taken by the test process' is limited to the generated package itself instead of all the packages needed to perform the generation. Further modified lexer-core to place the sets of tests for each variant grammar/lexer-action into a child process as well. This ensures that the memory needed by each variant grammar is limited to that child process, instead of accumulating in the controlling test process. __Attention__: This change requires a Kettle with support for `kt::sub`, added to Kettle with commit [ef384673c5] (2018-04-18 02:28:17). (user: aku, tags: mem-limit)
2018-04-17
20:36
[ad4f1d4287] Moved the main parts of `test/support/gen.tcl` into `bin/i-gen`. This new internal generator application uses the public `marpa-gen` as the underlying workhorse. The support code now invokes the internal generator instead of doing everything itself. With the operation confined to a child process the test process' amount of memory taken is limited to the generated package instead of also keeping all the overhead of generating Note, this does not help with the test suites based on lexer-core as that suite still loads/sources ten different lexer packages into the same process. Handling this requires more work, i.e. pushing the individual tests into their own child processes. (user: aku, tags: mem-limit)
2018-04-11
23:37 • Edit [302f47227a5f93b8|302f47227a]: Edit check-in comment. (user: aku)
23:36 • Edit [c3ff9015b635cfe8|c3ff9015b6]: Edit check-in comment. (user: aku)
23:33
[302f47227a] Extended the narrative tracing in `sem_tcl.c` to track refcounts. Found and fixed the cause for the orphaned Tcl_Obj*'s. The function `marpatcl_rtc_sv_astcl` did an extraneous refcount increment on the conversion result (SV tree to Tcl_Obj* tree). Parser operation based on RTC now does not leak anything anymore. (user: aku, tags: memory-cleanup)
07:43
[0d6e6c1dbe] Added code (sem_debug.c) to track SV allocation and release, and dump orphan SVs at the end. All SVs were orphaned, nothing released. Tracked to a bad guard condition in function `marpatcl_rtc_sv_unref`. As the code checks the refcount before decrementing it both 0 and 1 must trigger destruction, not just 0. Fixed. All SV structures are now properly released at the end (parser instance destruction). Still orphaned things left, these however are Tcl_Obj's. (user: aku, tags: memory-cleanup)
2018-04-10
04:55
[c3ff9015b6] container: Fixed leakage of priority and priority alternate objects. Test support extended with various diagnostic tools. Note, it looks like the RTC and glue into Tcl are also leaking like mad, especially around the semantic values and the (partial) ASTs we are constructing. Continue investigation and fixing. Started a new branch for this, (user: aku, tags: memory-cleanup)
2018-04-06
06:20
[065653b213] inbound, gate, lexer: Added a back-link from gate to inbound, analogous to the gate/lexer and lexer/parser connections. Dropped history management from the gate and changed its redo method to simply rewind the input instead of re-entering the tail end of the history. This is the first use of the new cursor movement methods added to inbound. Under the old regime using foreach, and steadily marching forward in the input all re-processing was done by remembering and recursively re-entering characters as needed, with additional loops at the various stages of the pipeline (`gate`). With the while-based cursor on the other hand we have only one (nested) processing loop (`inbound`), and all movement is handled by it. We cannot recurse, and we can't have a loops in `gate`. The nested loop in `inbound` is required to handle the case where we reach eof and the later stages then tell us `not yet`. The inner loop is the main processor, and the outer loop restarts it until eof actually suceeds. In `gate` the flush-signaling changed. In 'lexer' eof signaling to the `parser` is prevented when it bounces the input away from eof. (user: aku, tags: parse-events)
04:25
[cc42a78587] Pull gate readability changes into parse-event feature support. (user: aku, tags: parse-events)
2018-04-05
08:29
[0dff9cc32b] runtime-tcl, gate: Moved a few code blocks into their own methods to make their now-caller more readable. (user: aku, tags: trunk)
2018-04-04
17:57
[9c1fa89ac4] inbound: Converted the `foreach`-loop processing the physical input stream into a `while`. The location information now is a cursor into the input, instead of a dependent variable. This allows us to move backwards in the input, or ahead, as we see fit. Added movement methods (absolute, relative) for the cursor. No stream expansion yet, nor virtual streams. (user: aku, tags: parse-events)
16:58
[7693f8601c] Added generation of pre- and post-lexeme events. Extended the testsuite to demonstrate them. Note, the pre-lexeme trigger location does not match Marpa::R2 yet. (user: aku, tags: parse-events)
06:44
[97d4a5397c] Reworked the lexer somewhat. Moved match state into a nested object for easier access. Exposed to parse event handlers via a limiting facade. Outside entrypoint is parser method 'match', an ensemble. Moved to single event handler call bundling all relevant events. Internally also used to simplify GetSemanticValue. Updated discard event test. (user: aku, tags: parse-events)
2018-04-03
20:42
[da745b6e55] Pull recent fixes into the language work (user: aku, tags: parse-events)
20:37
[e5e442db2a] Fixed bug in the semantics' handling of :lexeme. Do not exclude the symbol from LATM fixup if the :lexeme adverbs do _not specify_ latm information. Facepalm. Found playing with lexeme events, and GC missing the crucial latm information, breaking generated test parsers. Updated tests to correct results. (user: aku, tags: trunk)
17:47
[7c7353ad24] Tests: Remove a leftover break from debugging something, wrongly committed. (user: andreask, tags: trunk)
04:42
[1c73889373] Get latest docs to work with. (user: aku, tags: parse-events)
04:41
[0f5c6931e2] Extended the Tcl runtime with basic support for events via callback. Set/unset/query, forwarding from the inner objects. Definition and storage of event maps. Proper pre-processing of such maps for the lexer, not yet for the parser. Added generation of discard events. Extended the testsuite to demonstrate the basic infrastructure, and discard events. Removal of trailing spaces. (user: aku, tags: parse-events)
2018-03-30
23:42
[d89aeafc1a] Leaf: Docs: More small fixes (user: aku, tags: docs)
23:38
[d41168399d] Get doc fixes (user: aku, tags: trunk)
23:37
[2a1acc9132] Docs: Typo fixes, phrasing fixes. (user: aku, tags: docs)
23:29 • Changes to wiki page Marpa (user: aku)
23:24
[607761c40d] Make recent documentation work official. (user: aku, tags: trunk)
23:23
[9f10a242a4] Docs work - Moved architecture from intro to dev guide. - Made intro a dispatcher to other documents based on readers goals. - Added reference for marpagen. - Added placeholder for SLIF, referencing the upstream Marpa::R2 documentation. (user: aku, tags: docs)
2018-03-29
23:14
[fdf9132f0d] Docs: Completed addition of feedback sections. Added audience/target information sections. (user: aku, tags: docs)
22:36
[2db7ae6cd3] Docs: Added changes, license documents, libmarpa requisite for installer, factored welcome message, added feedback in parts (user: aku, tags: docs)
22:05 • Changes to wiki page Marpa (user: aku)
21:46
[9ad599cfa5] Updated docs work with trunk changes. (user: aku, tags: docs)
05:34
[f99351071e] Started implementation of parse events. rt-Tcl first. Implemented generation from container, with fixes to have access to the stored G1 events. Extended testsuite to show event information, and updated older tests. Some whitespace corrections (removal of trailing spaces). Some tracing tag fixes. (user: aku, tags: parse-events)
2018-03-27
19:58
[1a2a8af59c] Leaf: Grammar edits: - Tweaked some of the formatting. - Main change: Redone the spine of the document structure. Moved the nullability around, enabling use of sequence rules. The price is paid by the paragraphs, having their separators in the AST (Cannot mask/hide the separator of a sequence from the AST). Explicit recursion for multiple separators in sequence however allows hiding that in a single separator. Still, AST nesting significantly reduced. Further doctoring is something for full custom command actions, or the semantics taking the AST. Regenerated parsers. Updated test suite to match. (user: aku, tags: language-doctools)
07:17
[de0a504dfd] Grammar: - Edited to provide the keywords with proper lexeme symbols instead of the ugly automatic names. - Reworked command definitions to enforce space after a command lead-in. - Added lots of custom actions (::first), to simplify the returned AST. Regenerated parsers. Went over test suite again. Removed the fail cases from Tcllib, completely bogus in places against the more strict specification. Reworked the ok cases and added the first proper ok results. Still thinking about the main spine of man pages and how to express nicely. The current explicit recursive structure nests a bit deep. (user: aku, tags: language-doctools)
2018-03-26
21:35
[d46d0a963a] Created parsers from draft. Made test suite functional. First runs, all results bad (different error messages on failure, different type of AST structure). (user: aku, tags: language-doctools)
21:08
[254c53ee1f] Documentation, installation guide: Added instructions for setting up `libmarpa`. (user: andreask, tags: trunk)
2018-03-25
05:15
[ecdb604ac6] Added untested draft grammar. (user: aku, tags: language-doctools)
02:53
[7e961f36fa] Added test-vectors used by Tcllib. (user: aku, tags: language-doctools)
2018-03-24
23:33
[26bb8496d5] Pull alias support into the example. (user: aku, tags: language-doctools)
23:31
[d60e61a1ce] Activated alias support in main. (user: aku, tags: trunk)
23:30
[85df278cb8] Closed-Leaf: marpa::unicode - Added alias handling. Updated testsuite. Further: - Reworked the table generator, more separate phases, less intertwined operations. Split across several files now, with each a set of related commands to manage part of the data structures. - BMP/SMP are directly generated as aliases where possible - Fixed issue with long-form category names for aliases. Tcl has its own definition of `control` (cc+cf+co). - Updated boot parsers. (user: aku, tags: cc-aliases)
04:23
[cb06be0e04] marpa::unicode - Added alias handling. Updated testsuite. Further: - Tweaked generator output. - Fixed issues with missing :bmp/:smp information for some aliases. (user: aku, tags: cc-aliases)
2018-03-23
23:59
[cc6426a1cc] Table generator: Convert `:bmp` and `:smp` CCs identical to their origin into aliases, reducing storage requirements. Output not usable anymore until the unicode layers gets extended to recognize and handle aliases. (user: andreask, tags: cc-aliases)
23:47
[7771b3d5ad] Introduced char class aliases into the output of the table generator. (user: andreask, tags: cc-aliases)
05:31
[ccb57223f3] Start on 2nd big example, doctools, of Tcllib (user: aku, tags: language-doctools)