View Ticket
Not logged in
Ticket UUID: 54329e39c757ee2aac298a142ecfe2a7ebe72391
Title: Compiled foreach uses excessive memory for lseq sequences
Type: Bug Version: 9.0
Submitter: apnadkarni Created on: 2023-07-11 10:55:35
Subsystem: - New Builtin Commands Assigned To: apnadkarni
Priority: 5 Medium Severity: Minor
Status: Closed Last Modified: 2023-07-12 07:40:09
Resolution: Fixed Closed By: jan.nijtmans
    Closed on: 2023-07-12 07:40:09
Description:
This is a memory usage issue, not bug. As shown below, a foreach loop over a lseq within a proc results in a loss of the memory efficiency benefit of lseq.

```
% set l [lseq 1000000]; llength $l
1000000
% proc getmem {} {set fd [open /proc/[pid]/statm]; return [read $fd][close $fd]}
% getmem
6026 1777 1468 558 0 412 0

% foreach x $l {}
% getmem
6026 1777 1468 558 0 412 0

% proc p l {foreach x $l {}}
% p $l
% getmem
17731 13556 1477 558 0 12117 0
```

Notice the jump in memory usage after the loop in the proc. The same loop on the command line does not increase memory.

I suspect the cause is the use of Tcl_ListObjGetElements somewhere in the execution engine in lieu of Tcl_ListObjIndex.
User Comments: jan.nijtmans added on 2023-07-12 07:40:09:

Many thanks, Ashok!


apnadkarni added on 2023-07-12 06:44:56:
Fixed for [9.0](https://core.tcl-lang.org/tcl/info/3435cde0f55776b4) and [8.7](https://core.tcl-lang.org/tcl/info/95cab70fe8b80e1c).

apnadkarni added on 2023-07-11 16:41:07:
Was in the process of doing that distinction. There is a some measurable difference for "traditional" lists between the getelements and index. A little surprising to me because the getelements there happens on every iteration just like the index. On the other hand, the former is a macro and the latter a function so that might be something I look at later.

apnadkarni added on 2023-07-11 16:17:15:
I already had inserted the CACHE macros and was coding the change to distinguish between abstract and non-abstract probably as you were updating the ticket :-)

The original is more efficient for non-abstract lists though I am a bit surprised at the difference. Code-wise there is very little difference calling Tcl_ListObjIndex noting that the GetElements in that fragment is done on *every* iteration. But its measurable, may be some cache effects? Who knows.

Still doing performance measurements.

/Ashok

griffin added on 2023-07-11 15:34:44:
The proposed fix looks good.  
I wonder if it is worthwhile to have separate loops for Lists vs Abstract Lists, as GetElements is more efficient for Lists. It may be that the difference is not worth the complexity. IDK.

Note that there are a number of places where DECACHE/CACHE calls need to be inserted in TBCE. I've done that in tip-636 (main). [tclExecute.c](https://core.tcl-lang.org/tcl/fdiff?v1=b7ee7c30a6f02625&v2=5d734646930c5089)

jan.nijtmans added on 2023-07-11 12:14:16:

Just one remark: I guess Tcl 8.7 has the same problem. So I recommend backporting this (which can be done after merging this to trunk)

+1


apnadkarni added on 2023-07-11 11:36:54:
Proposed fix in branch bug-54329e39c7. Needs review.