Marpa

Timeline
Login
Tcl 2016 Conference, Houston/TX, US, Nov 14-18
Send your abstracts to tclconference@googlegroups.com by Sep 12.

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

50 most recent timeline items

2018-07-17
05:29
[dd2dfa9d07] Leaf: Filled in most of the facade. Still missing parts relevant to before/after events. Location accessors implemented, only correct for ASCII. UTF support is still to do. Moved the API functions into inbound and lexer, with pieces in the pedesc class. Dropped the separate pedesc header and C sources. Optimized moveto, add the delta to the positions before delivery to the engine. Lexer now manages all the new fields for the match state (initialization, update, reset). lexeme data for SV now coming out of the new fields and accessors. Event testing now recording any error we may receive from the match facade. (user: aku, tags: parse-events-rtc)
2018-07-13
23:47
[96dc913d9e] Attention: This commit will likely not even compile. It is saved scratch state of work on the rtc lexer match state to complete the parse event facade. Reworking the lexer state internals for cached access to the information, ability to modify. This will affect the lexer/parser interface, namely the transmission of found symbols and semantic values. (user: andreask, tags: parse-events-rtc)
22:27
[91aa29f8f2] Added tracing of the new event matching and reporting code. Tweaked tracing of the EH functions (separate stream). Added detection and reporting of discard events. Fixed generated lexer event entries, engine operates with the ACS symbols. Updated tests. Discard events are detected and reported. Testsuite fails because the PE descriptor facade is not completely filled out yet, and the incomplete methods have divergent signatures from the expected. While the Tcl errors thrown by the event recorder callback used in the testsuite are ignored during execution, they are properly seen in the narrative tracing proving that the discard callbacks works. Fill out structure and facade before implementing the before/after callbacks. (user: andreask, tags: parse-events-rtc)
18:06
[c20d03a3ef] Linked the PE descriptor facade into the lexer/parser templates. Removed scratch notes from the facade. Updated tests. (user: andreask, tags: parse-events-rtc)
2018-07-12
21:35
[9e2ca3d617] Extended the facade with set/get for the class rtc variable needed by the constructor (user: andreask, tags: parse-events-rtc)
20:58
[84fa7104f2] Started work on generic parse event descriptor access. Using a critcl::class as facade to the structures, ensemblified methods. Requires critcl::class 1.1 to disable tcl-api, generate c-api. (user: andreask, tags: parse-events-rtc)
20:01
[915f151c40] Fixed typo in comment (user: andreask, tags: parse-events-rtc)
16:41 • Edit [f5e6063aeb9c9f52|f5e6063aeb]: Edit check-in comment. (user: aku)
05:43
[66339c7f20] Fixed [f5e6063aeb] memory smash. Miscounted references to the `self` argument of parse event callbacks. Updated tests with all the generator changes. rtc-runtime tests: L0 parse event cases failing as expected, as event generation is not done yet. However we are now at the point where we can start on adding this in. Most of the foundations are now present. Notably still missing are the parse event descriptor structures and its linkage to the lexer state. (user: aku, tags: parse-events-rtc)
2018-07-11
22:52
[f5e6063aeb] Filled in the marpatcl_rtc_eh_... functions and structures. Fixed bogus declaration syntax for the generated event structures. __Attention__: Seg.fault in core. Likely caused by the new functions, structures, and their use. Update: Fixed with [66339c7f20]. (user: andreask, tags: parse-events-rtc)
20:28
[64a75ee092] Follow up to lex-only refactoring, updated tests. (user: andreask, tags: parse-events-rtc)
20:03 • Edit [1d1b5cb7d865dd1a|1d1b5cb7d8]: Edit check-in comment. (user: aku)
19:59 • Edit [3df1d4cea4747090|3df1d4cea4]: Edit check-in comment. (user: aku)
19:58
[fce9c19274] Ripped the general structures and code for lex-only token/value handling out of the clex template and placed them into the marpa::runtime::c package for sharing. As part of that the result and event callbacks from RTC now have separate client data information. Note, the yet undefined `eh` structures and functions already referenced by the event handling code are a similar thing for that aspect. (user: andreask, tags: parse-events-rtc)
19:47
[f3f39059c2] Always initialize variable (user: andreask, tags: parse-events-rtc)
19:46
[8202ba0482] Added notes about uninitialized memory to the two set implementations. (user: andreask, tags: parse-events-rtc)
19:43
[1d1b5cb7d8] Memory smash fixed. SV ref miscount in the lexer in lex-only mode when one SV is re-used for multiple tokens in the same location. Win for the (SEM_)REF_DEBUG functionality coming out of the mem-limit, memory-cleanup branches. Valgrind was no help. (user: andreask, tags: parse-events-rtc)
04:36
[afeb1c4ea6] Fixed missing cleanup of callback field, caused a bogus 2nd destroy on object destruction. Tweaked comment in test support code. (user: aku, tags: parse-events-rtc)
00:07
[3df1d4cea4] Filling in the C-level data structures and API changes implied by the template changes for clex and cparse generators. ATTENTION: Manually modified the builtin parsers (slif, literals) to match the changed structures and function signatures. ATTENTION: Memory smash somewhere in the zeta-rtc-lexer tests. (Fixed with commit [1d1b5cb7d8]) (user: andreask, tags: parse-events-rtc)
2018-07-10
23:13
[e274d2eda6] Show i-gen critcl command in the log. Added main tclsh include to the set of include paths to search. (user: aku, tags: parse-events-rtc)
22:21
[c230bc5be1] Converted the eof/enter callbacks of clex to critcl::callback. (user: andreask, tags: parse-events-rtc)
21:13
[c7e33a1f26] Extended asset management to allow more than one asset. Generate C event datastructures, lex & parse. Updated and extended tests. (user: andreask, tags: parse-events-rtc)
2018-06-26
18:20
[22e19935f3] Leaf: Make rewind more robust. Tweak post-event input movement. Added event scratch docs (user: andreask, tags: parse-events)
2018-04-27
15:48 • Edit [714d4386030cdf8d|714d438603]: Edit check-in comment. (user: aku)
2018-04-20
21:28
[faebf8fba4] inbound: Absolute movement, tweaked for consistency. Extended with optional delta arguments. gate: Extended with forwards to the input location accessor and modifiers. This provides parse event handlers with the ability to move in the input. lexer: Moved redo to before we handle parse events. This enables parse event handlers to modify the location without interference from the system itself. Modified pre-lexeme event generation to move input to the start of the lexeme. Fixed pe-fill method, forgot to set flag for the incremental rebuild of the symbol/sv tables. parse descriptor: See gate. Further fixed accessor setup. Extended view to report input location. testsuite: Report input location. Move input location to the end of the lexeme. Needed now that pre-lexeme events have the location at the start. Updated expected results. (user: aku, tags: parse-events)
04:33
[07ca2e2be0] Pulled memory fixes into the feature branch implementing parse events. (user: aku, tags: parse-events)
2018-04-19
19:49 • Edit [cc42a785875a7adf|cc42a78587]: Edit check-in comment. (user: aku)
19:47
[dcf340cd32] Leaf: Update main line with the accumulated memory fixes. (user: aku, tags: trunk)
19:46 • Edit [a29c4613b0ad2de3|a29c4613b0]: Mark "Closed". (user: aku)
19:44
[dc34e1afbb] Closed-Leaf: Merged series of fixes for memory issues (memory leaks, management mismatches, ...). Together with the push of memory-intensive tests into child processes done here the testsuite should now be properly constrained again wrt memory usage. Updated tests. (user: aku, tags: mem-limit)
18:18
[a29c4613b0] Closed-Leaf: Added test against unbound parser memory usage in parser-core. Found and fixed SV ref-count mismanagement (RCM) in the lex-only code path of the RTC. Added narrative tracing to the code path. Cleanup of lexer-core testsuite with regard to memory debugging. Found and fixed Tcl_Obj* RCM in the lex-only critcl template. Added narrative tracing to the template. (user: aku, tags: memory-cleanup)
2018-04-18
23:41
[714d438603] Do not mix malloc and Tcl allocation routines. Even when the malloc is hidden, here in `strdup`. Used to allocate strings with strdup, then release with ckfree/Tcl_Free. This messed up memory mgmt internals to the point of memory smashes. Fixed, now using our own implementation of strdup based on the allocation macros from environment.h. This ensures that alloc and free match. Thank you, valgrind. (user: aku, tags: memory-cleanup)
03:08
[ddc1e67640] Moved the main parts of `test/support/gen.tcl` into `bin/i-gen`. This new internal generator application uses the public `marpa-gen` as the underlying workhorse. The support code now invokes the internal generator instead of doing everything itself. With the operation confined to a child process the amount of memory taken by the test process' is limited to the generated package itself instead of all the packages needed to perform the generation. Further modified lexer-core to place the sets of tests for each variant grammar/lexer-action into a child process as well. This ensures that the memory needed by each variant grammar is limited to that child process, instead of accumulating in the controlling test process. __Attention__: This change requires a Kettle with support for `kt::sub`, added to Kettle with commit [ef384673c5] (2018-04-18 02:28:17). (user: aku, tags: mem-limit)
2018-04-17
20:36
[ad4f1d4287] Moved the main parts of `test/support/gen.tcl` into `bin/i-gen`. This new internal generator application uses the public `marpa-gen` as the underlying workhorse. The support code now invokes the internal generator instead of doing everything itself. With the operation confined to a child process the test process' amount of memory taken is limited to the generated package instead of also keeping all the overhead of generating Note, this does not help with the test suites based on lexer-core as that suite still loads/sources ten different lexer packages into the same process. Handling this requires more work, i.e. pushing the individual tests into their own child processes. (user: aku, tags: mem-limit)
2018-04-11
23:37 • Edit [302f47227a5f93b8|302f47227a]: Edit check-in comment. (user: aku)
23:36 • Edit [c3ff9015b635cfe8|c3ff9015b6]: Edit check-in comment. (user: aku)
23:33
[302f47227a] Extended the narrative tracing in `sem_tcl.c` to track refcounts. Found and fixed the cause for the orphaned Tcl_Obj*'s. The function `marpatcl_rtc_sv_astcl` did an extraneous refcount increment on the conversion result (SV tree to Tcl_Obj* tree). Parser operation based on RTC now does not leak anything anymore. (user: aku, tags: memory-cleanup)
07:43
[0d6e6c1dbe] Added code (sem_debug.c) to track SV allocation and release, and dump orphan SVs at the end. All SVs were orphaned, nothing released. Tracked to a bad guard condition in function `marpatcl_rtc_sv_unref`. As the code checks the refcount before decrementing it both 0 and 1 must trigger destruction, not just 0. Fixed. All SV structures are now properly released at the end (parser instance destruction). Still orphaned things left, these however are Tcl_Obj's. (user: aku, tags: memory-cleanup)
2018-04-10
04:55
[c3ff9015b6] container: Fixed leakage of priority and priority alternate objects. Test support extended with various diagnostic tools. Note, it looks like the RTC and glue into Tcl are also leaking like mad, especially around the semantic values and the (partial) ASTs we are constructing. Continue investigation and fixing. Started a new branch for this, (user: aku, tags: memory-cleanup)
2018-04-06
06:20
[065653b213] inbound, gate, lexer: Added a back-link from gate to inbound, analogous to the gate/lexer and lexer/parser connections. Dropped history management from the gate and changed its redo method to simply rewind the input instead of re-entering the tail end of the history. This is the first use of the new cursor movement methods added to inbound. Under the old regime using foreach, and steadily marching forward in the input all re-processing was done by remembering and recursively re-entering characters as needed, with additional loops at the various stages of the pipeline (`gate`). With the while-based cursor on the other hand we have only one (nested) processing loop (`inbound`), and all movement is handled by it. We cannot recurse, and we can't have a loops in `gate`. The nested loop in `inbound` is required to handle the case where we reach eof and the later stages then tell us `not yet`. The inner loop is the main processor, and the outer loop restarts it until eof actually suceeds. In `gate` the flush-signaling changed. In 'lexer' eof signaling to the `parser` is prevented when it bounces the input away from eof. (user: aku, tags: parse-events)
04:25
[cc42a78587] Pull gate readability changes into parse-event feature support. (user: aku, tags: parse-events)
2018-04-05
08:29
[0dff9cc32b] runtime-tcl, gate: Moved a few code blocks into their own methods to make their now-caller more readable. (user: aku, tags: trunk)
2018-04-04
17:57
[9c1fa89ac4] inbound: Converted the `foreach`-loop processing the physical input stream into a `while`. The location information now is a cursor into the input, instead of a dependent variable. This allows us to move backwards in the input, or ahead, as we see fit. Added movement methods (absolute, relative) for the cursor. No stream expansion yet, nor virtual streams. (user: aku, tags: parse-events)
16:58
[7693f8601c] Added generation of pre- and post-lexeme events. Extended the testsuite to demonstrate them. Note, the pre-lexeme trigger location does not match Marpa::R2 yet. (user: aku, tags: parse-events)
06:44
[97d4a5397c] Reworked the lexer somewhat. Moved match state into a nested object for easier access. Exposed to parse event handlers via a limiting facade. Outside entrypoint is parser method 'match', an ensemble. Moved to single event handler call bundling all relevant events. Internally also used to simplify GetSemanticValue. Updated discard event test. (user: aku, tags: parse-events)
2018-04-03
20:42
[da745b6e55] Pull recent fixes into the language work (user: aku, tags: parse-events)
20:37
[e5e442db2a] Fixed bug in the semantics' handling of :lexeme. Do not exclude the symbol from LATM fixup if the :lexeme adverbs do _not specify_ latm information. Facepalm. Found playing with lexeme events, and GC missing the crucial latm information, breaking generated test parsers. Updated tests to correct results. (user: aku, tags: trunk)
17:47
[7c7353ad24] Tests: Remove a leftover break from debugging something, wrongly committed. (user: andreask, tags: trunk)
04:42
[1c73889373] Get latest docs to work with. (user: aku, tags: parse-events)
04:41
[0f5c6931e2] Extended the Tcl runtime with basic support for events via callback. Set/unset/query, forwarding from the inner objects. Definition and storage of event maps. Proper pre-processing of such maps for the lexer, not yet for the parser. Added generation of discard events. Extended the testsuite to demonstrate the basic infrastructure, and discard events. Removal of trailing spaces. (user: aku, tags: parse-events)