View Ticket
Not logged in
Ticket UUID: f586089a2b0714405275c3cf8604e784a9c5641e
Title: Test failures due to macOS_hangs merge
Type: Bug Version: 8.6
Submitter: jan.nijtmans Created on: 2020-07-21 13:09:00
Subsystem: 01. Notifier Assigned To: marc_culler
Priority: 5 Medium Severity: Important
Status: Closed Last Modified: 2020-08-26 15:00:42
Resolution: Fixed Closed By: jan.nijtmans
    Closed on: 2020-08-26 15:00:42
Description:

Since the macOS_hangs branch is merged to core-8-6-branch, there are Travis build-failures. Testcases sensible are chan-io-50.x, io-50.x and eventh-1.1: those fail sometimes randomly (maybe one out of 50 times) but this is just enough to cause not a single build success since then.

I don't suspect that the macOS notifier code is wrong, it might as well be that the test-cases are too sensible to the exact order of events as they happem. I see the same failures sometimes on Cygwin: The cygwin notifier handles 2 event-loops as well, the Win32 one and its own, just as MacOS handled the Apple event loop besides its own.

Since those test-failures don't appear to be a problem in real-life (I don't encounter problematic behavior in normal use at all!) I now marked those test-cases as "nonPortable". But it would be nice if this could be fixed before the release of 8.6.11

User Comments: jan.nijtmans added on 2020-08-26 15:00:42:

Travis failed again in chan-io-50.1. See: https://travis-ci.org/github/tcltk/tcl/jobs/721058471

Follow-up commit here: [be456e2d1bda92bc]. This is actually the same change as done in testcase io-50.1.


marc_culler (claiming to be Marc Culler) added on 2020-08-11 17:29:05:
Travis passed today, so I will close this ticket now.

jan.nijtmans added on 2020-08-10 15:36:44:

Thanks Marc, looks good!


marc_culler (claiming to be Marc Culler) added on 2020-08-10 12:59:30:

After checkin 3848f45fbc all tests passed with all of the XCode versions available on Travis CI. So I merged the bugfix branch into core-8-6-branch. I will close this ticket if the tests continue to pass.


marc_culler (claiming to be Marc Culler) added on 2020-07-24 12:16:18:
So it looks like using the formerly deprecated but now magically restored OSSpinlock instead of the newer more efficient os_unfair_lock allows the
tests to pass in 4 out of 6 versions of XCode on Travis.

I am sure these tests rely on how a race condition resolves.

marc_culler (claiming to be Marc Culler) added on 2020-07-22 13:58:34:
One change in tclMacOSNotifier.c, which was completely independent from
everything else, was to switch from OSSpinLock to os_unfair_lock.  I switched
that back to see if it makes any difference on Travis.  The original reason
for making the change was that OSSpinLock had been deprecated since 10.13.
But, when I switched back to OSSpinLock and built Tcl on 10.15 and 10.16
(== 11.1 ??)  I noticed that the deprecation warnings no longer appeared.

We all know that Apple software engineers are nearly infallible (i.e. that
Apple rarely acknowledges their bugs) so I think that the quiet disappearance
of the deprecation warning may be as close as we will get to learning that
there are problems with os_unfair_lock.

marc_culler (claiming to be Marc Culler) added on 2020-07-22 13:37:24:
Is there any way to trigger a build for a particular commit?  Or are we
back in the good old days of batch computing, submitting our deck of
punchcards and coming back tomorrow to look for the output.

We are debugging Travis itself, after all, since none of these symptoms
have ever been seen on any other platform.

jan.nijtmans added on 2020-07-22 07:38:41:

The sync with GIT is done only once a day. The branch is triggered now, the first build will start soon today


marc_culler (claiming to be Marc Culler) added on 2020-07-22 00:54:00:
When I go to https://travis-ci.org/github/tcltk/tcl/branches I do not see
bug-f586089a2b listed as a branch.  How can I view builds for that branch?

jan.nijtmans added on 2020-07-21 23:16:12:
> Here is my first guess about why these tests behave ...

Sound reasonable. So, let's rewrite/adapt those testcases to make them reliable. Until then, they are marked "nonPortable".

I created the [bug-f586089a2b] branch meant for this work.

jan.nijtmans added on 2020-07-21 23:10:46:

>Do we know anything about the Travis VM used for macOS testing?

Yes: https://docs.travis-ci.com/user/reference/osx/


marc_culler (claiming to be Marc Culler) added on 2020-07-21 20:13:59:
Here is my first guess about why these tests behave in an indeterminate way:
The test is placing bets at the racetrack and it only passes if it wins.

The mac notifier design includes a thread which runs select and queues file
events when select reports a change in status for one of the files it is
watching.

I see test failures in Travis that look like this:

--- Result was:

{del calling recursive} {del after update} {notcalled was called!! file8}
{del deleted notcalled} {del deleted myself}

---- Result should have been (exact matching):

{del calling recursive} {del deleted notcalled} {del deleted myself}
{del after update}

I think this test failure is reporting that the call to update returned before
notcalled was deleted, while the test expected it to return after.  The random
behavior would seem to indicate that sometimes update returns before notcalled
is deleted and sometimes it returns afterward.  Given that Tcl_WaitForEvent
used to do a lot more work, since it would call Tcl_ServiceAll while it was
supposedly waiting, I think it is not surprising that update might return
earlier than it used to.

But I think the bottom line is that we have two threads racing to complete
different tasks.  The test passes if one thread wins the race and it fails
if the other one wins.  I don't see how a test like that could ever be
reliable.

marc_culler (claiming to be Marc Culler) added on 2020-07-21 16:43:42:
Do we know anything about the Travis VM used for macOS testing?