In core-8-branch, after building Tcl and installing it:

% package require msgcat
couldn't read file "C:/Users/francois/Documents/Development/tcltk-fossil/tcltk/lib/tcl8.7/msgcat/msgcat.tcl": no such file or directory
%

Happens at least on Windows and macOS.

In contrast, core-8-6-branch provides the expected answer ("1.6.1").

Since fix is now applied to core-8-branch and later, closing.

Zipfs builds are generating a file system that is only readable from the version of Tcl that is running. We have our little "museum of Tcl" because in an install the latest Tcl is competing in a file system with whatever Tcls may have come before. Breaking /library up into chunks for zipfs builds is an unneeded complication.

In testing zipfs was causing issues with safe Tcl interpreters because (depending on environment) there was some install foolery afoot to obscure core packages that were not Safe Tcl worthy. I assembled the spirit of those various install mechanism into a manifest file that tells the interpreter, for clear and for certain, which packages may be loaded in a normal interpreter and which packages may be loaded in a safe interpreter.

Clearly the end of the init.tcl file was not the place for that because of the issue raised in this ticket.

The solution I've come up with is to take what was in that manifest and move it out of init.tcl and into a file the installers and Tcl will ignore (but still maintained in the /library file system via scm) as manifest.txt.

When building zipfs file systems, we copy /library into the VFS. Now there will be one extra step, after the copy is completed, to rename manifest.txt to pkgIndex.tcl. Because tcl_library is already in the auto_path of an unmodified Tcl interpreter, all of the packages we are looking to expose (either in a "real" interp or a "safe" interp) are not exposed in a way that make sense.

And in the case of Nmake, you can continue to use your current install procedure and the manifest.txt file will simply be ignored. Though it would be nice to bring nmake up to speed on how to wrap /library into a vfs that is attached to the tcl.dll file.

I see that we are deputizing Tcl for a couple of other steps in the nmake build process. Perhaps we could use the zip encoded in the core now to do the wrapping? What other platforms are doing is copying /library to libtcl.vfs, and then using a native zip program to build tclVERSION.zip.

For static libraries, this zip file is copied next to the libtclVERSION.a file. For dynamic builds, tclVERSION.zip is concatenated onto the end of the libtclVERSION.(so|dll|dylib), and then we perform a call to zip to cleanup the offset.

Inside the tip430 implementation we have a zip encoder that could do this entire process in one step

I have put together a wrapper tcl script (checked into [core-8-branch]) and a provisional patch to makefile.vc to utilize it (in the [nmake-zipfs] branch). I'm a little rusty on my visual studio, so if Jan or fvogel could try that branch out that would be great.

The patch eliminates installing files and instead wraps /library in a zipfs system attached to the tcl.dll (for dynamic builds) or tcl$(VERSION).zip, which is dropped in next to the tclsh.exe during install. I think I got it right, but it never hurts to test. I have tested that the wrapper script does indeed work on Mac.

--Sean

Thanks, hypnotoad, for the upcoming Tcl 8.7 alpha, this solutions will do!

The thing missing in TIP #430 is handling of TM's (Tcl Modules). The TIP specifies that packages can have their files added to ZIPFS_ROOT/lib/PGKNAME, which is automatically found by tcl_findLibrary(). But it doesn't specify how Tcl Modules can be put into a ZIP, and found as well. That's what the difference in handling explains.

So we should specify where TM's should land in ZIP files, and adapt the TM machinery to handle that. I would propose "ZIPFS_ROOT/tcl8/8.x".

Um, guys, that "problem" in init.tcl is bootstrap for the library file system in zipfs enabled builds. I have to admit, when tip430 was formulated I ran out of steam when it came to hacking the installation process under Nmake for windows. But sawing the problem out of init.tcl is going to screw up every non-Windows on nmake users.

I am working up a fix that init.tcl will be only present a boot strap for installations where /library was packaged as either part of the executable, packed in the dll, or is expected to be loaded from a zip file.

I'm baking Jan's fix so far out to a branch until we hammer the issue out to the satisfaction of all.

--Sean "The Hypnotoad" Woods

Thanks for the fix!

I wanted to confirm but core-8-branch currently does not build on Windows.

tclOOCall.c
C:\Users\francois\Documents\Development\tcltk-fossil\tcl\win\..\generic\tclOOCall.c(2037) : error C2220: warning treated as error - no 'object' file generated
C:\Users\francois\Documents\Development\tcltk-fossil\tcl\win\..\generic\tclOOCall.c(2037) : warning C4028: formal parameter 1 different from declaration
C:\Users\francois\Documents\Development\tcltk-fossil\tcl\win\..\generic\tclOOCall.c(2037) : warning C4028: formal parameter 2 different from declaration
C:\Users\francois\Documents\Development\tcltk-fossil\tcl\win\..\generic\tclOOCall.c(2037) : warning C4028: formal parameter 4 different from declaration

The actual problem turned out to be in init.tcl: http and msgcat are NOT installed in $tcl_library, so they shouldn't be handled especially, as mentioned in the comment: "Package manifest for all Tcl packages included in the /library file system"

It's pointless to fix something that's going away. Complete the implementation of the replacement.

Donal,

currently, the TCL module load mechanism is broken. The same issue applies for the http package.

Thank you, Harald

TIP 430 has been approved. The release_zipfs branch is the implementation of TIP 430 for Tk.

This bug reports that the msgcat package cannot be found, or an installation is found that is broken. After TIP 430, a non-broken msgcat package is built into wish8.7 itself. That's the simplest, already approved path to ending the misbehavior reported.

Oh, that branch is in the Tk repo, not Tcl! Thank you for looking at this issue.

Not sure what you're asking to test here exactly. The release_zipfs branch is off core-8-6-branch of Tk. The described issue happens in Tcl (without Tk) and the msgcat/http packages it provides. What is the link to Tk here? What combination of Tcl and Tk branches are you requiring to be tested?

Added a new branch:

release_zipfs: [f38b6733]

See if this fixes things.

Ignore that last comment... I have to rebuild the entire patch from scratch. There is no clear migration path from the Tk that patch originally branched off from back to 8.6.9 without taking us through trunk.

Do these problems go away when you build from the core-zip-vfs branch?

I can confirm the bug on windows with current core-8-branch of tcl "3f67becca2" dated 2018-11-15.

The msgcat module file is in lib/tcl/8.7:

C:\test\tcl87a1_3f67becca2_2018-11-15\lib\tcl8\8.7\msgcat-1.7.0.tm

Here is a log:

% package require msgcat
couldn't read file "C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8.7/msgcat/msgcat.tcl": no such file or directory
% package require http
couldn't read file "C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8.7/http/http.tcl": no such file or directory
% ::tcl::tm::path list
C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/site-tcl C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.0 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.1 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.2 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.3 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.4 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.5 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.6 C:/test/tcl87a1_3f67becca2_2018-11-15/lib/tcl8/8.7

So, apparently, there is an issue in the module load code: The TCL modules are found, but the load is tried in a file: "<Module path>/<Module name>/<Module name>.tcl"

Thank you, Harald

Assigning to hypnotoad. Looks like the best solution is to complete TIP 430 Implementation in Tk.

Thank you for the added detail.

As it happens, we should be able to do even better now. Tcl 8.7 includes TIP 430, so now there should be no need to go searching anywhere for msgcat. It should be in the VFS appended to the tclsh executable itself.

Build and install Tcl 8.7 and Tk 8.7 like this:

$ rm -rf build
$ cd tcl
$ fossil update core-8-branch ; make -C macosx ; sudo make -C macosx install
$ cd ../tk
$ fossil update trunk ; make -C macosx ; sudo make -C macosx install

Now run Wish:

$ Wish
% package require msgcat
couldn't read file "/Library/Frameworks/Tcl.framework/Versions/8.7/Resources/Scripts/msgcat/msgcat.tcl": no such file or directory

If you look for msgcat you find a .tm file:

$ find /Library/Frameworks/Tcl.framework/Versions/8.7 -name 'msgcat*'
/Library/Frameworks/Tcl.framework/Versions/8.7/Resources/tcl8/8.7/msgcat-1.7.0.tm

So, yes, it is being installed as a TM, whatever that is, but Tcl 8.7 does not seem to know how to load a TM.

You can make the error go away by doing this:

$ sudo mkdir /Library/Frameworks/Tcl.framework/Versions/8.7/Resources/Scripts/msgcat
$ sudo cp /Library/Frameworks/Tcl.framework/Versions/8.7/Resources/tcl8/8.7/msgcat-1.7.0.tm /Library/Frameworks/Tcl.framework/Versions/8.7/Resources/Scripts/msgcat/msgcat.tcl

Then Wish is able to load the package:

$ Wish
% package require msgcat
1.7.0

For Winwows: Build with MSVC (using makefile.vc) and install alike, basically:

    nmake -f makefile.vc realclean OPTS="symbols" STATS="none"
    nmake -f makefile.vc release OPTS="symbols" STATS="none"
    nmake -f makefile.vc install INSTALLDIR=%MYTCLTK% OPTS="symbols" STATS="none"

For macOS this was reported to me by Marc Culler. I just tried but can't reproduce right now on this platform.

This didn't happen in the (perhaps moderately distant) past.

I suspect (but didn't check this in practice) https://core.tcl.tk/tcl/timeline?r=tip499-msgcat-custom-preferences and/or https://core.tcl.tk/tcl/timeline?r=tip490-msgcat-oo-2 to have forgotten some platforms.

(BTW: TM stands for...?)

"building Tcl and installing it"

  What build system is in use when this is broken? Looks like something is installing the wrong files. msgcat is supposed to be installed as a TM.