Fossil

Check-in [36620b74a4]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Modify the out-of-order card detection in the structural artifact parser to allow N and P cards of a technote to be in the wrong order. [15d04de574383d61|Ticket 15d04de574383d61].
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | manifest-sort-check
Files: files | file ages | folders
SHA3-256: 36620b74a46b2073122a0a1e17dca06d1de378e4303f5f900b3ab4ba72ec347a
User & Date: drh 2021-02-10 23:34:09.068
Context
2021-02-10
23:49
Give a better error message if structural artifact cards are out-of-order. Closed-Leaf check-in: 72f5101ef0 user: drh tags: manifest-sort-check
23:34
Modify the out-of-order card detection in the structural artifact parser to allow N and P cards of a technote to be in the wrong order. [15d04de574383d61|Ticket 15d04de574383d61]. check-in: 36620b74a4 user: drh tags: manifest-sort-check
22:13
This seems to fix the missing is-sorted check in manifest_parse(), but (A) it needs more testing and (B) could use a better error message than the one it ends up generating. Edit: WARNING: fixing this will cause control artifacts which previously passed (but should not have) to fail. check-in: 2e7a0aacdb user: stephan tags: manifest-sort-check
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/manifest.c.
532
533
534
535
536
537
538
539










540
541
542
543
544
545
546
  pContent = &p->content;

  /* Begin parsing, card by card.
  */
  x.z = z;
  x.zEnd = &z[n];
  x.atEol = 1;
  while( (cType = next_card(&x))!=0 && cType>=cPrevType ){










    lineNo++;
    if( cType<'A' || cType>'Z' ) SYNTAX("bad card type");
    seenCard |= 1 << (cType-'A');
    cPrevType = cType;
    switch( cType ){
      /*
      **     A <filename> <target> ?<source>?







|
>
>
>
>
>
>
>
>
>
>







532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
  pContent = &p->content;

  /* Begin parsing, card by card.
  */
  x.z = z;
  x.zEnd = &z[n];
  x.atEol = 1;
  while( (cType = next_card(&x))!=0 ){
    if( cType<cPrevType ){
      /* Cards must be in increasing order.  However, out-of-order detection
      ** was broken prior to 2021-02-10 due to a bug.  Furthermore, there
      ** was a bug in technote generation (prior to 2021-02-10) that caused
      ** the P card to occur before the N card.  Hence, for historical
      ** compatibility, we do allow the N card of a technote to occur after
      ** the P card.  See tickets 15d04de574383d61 and 5e67a7f4041a36ad.
      */
      if( cType!='N' || cPrevType!='P' || p->zEventId==0 ) break;
    }
    lineNo++;
    if( cType<'A' || cType>'Z' ) SYNTAX("bad card type");
    seenCard |= 1 << (cType-'A');
    cPrevType = cType;
    switch( cType ){
      /*
      **     A <filename> <target> ?<source>?
Changes to www/fileformat.wiki.
84
85
86
87
88
89
90


91
92
93
94
95
96
97
98
(ASCII: 0x0a) character. Each card begins with a single
character "card type".  Zero or more arguments may follow
the card type.  All arguments are separated from each other
and from the card-type character by a single space
character.  There is no surplus white space between arguments
and no leading or trailing whitespace except for the newline
character that acts as the card separator.  All cards must be in strict


lexicographical order.  There may not be any duplicate cards.

In the current implementation (as of 2017-02-27) the artifacts that
make up a fossil repository are stored as delta- and zlib-compressed
blobs in an <a href="http://www.sqlite.org/">SQLite</a> database.  This
is an implementation detail and might change in a future release.  For
the purpose of this article "file format" means the format of the artifacts,
not how the artifacts are stored on disk.  It is the artifact format that







>
>
|







84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
(ASCII: 0x0a) character. Each card begins with a single
character "card type".  Zero or more arguments may follow
the card type.  All arguments are separated from each other
and from the card-type character by a single space
character.  There is no surplus white space between arguments
and no leading or trailing whitespace except for the newline
character that acts as the card separator.  All cards must be in strict
lexicographical order (except for an
[./fileformat.wiki#outofordercards|historical bug compatibility]).
There may not be any duplicate cards.

In the current implementation (as of 2017-02-27) the artifacts that
make up a fossil repository are stored as delta- and zlib-compressed
blobs in an <a href="http://www.sqlite.org/">SQLite</a> database.  This
is an implementation detail and might change in a future release.  For
the purpose of this article "file format" means the format of the artifacts,
not how the artifacts are stored on disk.  It is the artifact format that
871
872
873
874
875
876
877







































878
879
880
881
882
883
884
885

<a name="addenda"></a>
<h2>4.0 Addenda</h2>

This section contains additional information which may be useful when
implementing algorithms described above.








































<h3>4.1 R-Card Hash Calculation</h3>

Given a manifest file named <tt>MF</tt>, the following Bash shell code
demonstrates how to compute the value of the <b>R</b> card in that manifest.
This example uses manifest [28987096ac]. Lines starting with <tt>#</tt> are
shell input and other lines are output. This demonstration assumes that the
file versions represented by the input manifest are checked out
under the current directory.







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|







873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926

<a name="addenda"></a>
<h2>4.0 Addenda</h2>

This section contains additional information which may be useful when
implementing algorithms described above.

<a name="outofordercards"></a>
<h3>4.1 Relaxed Card Ordering Due To An Historical Bug</h3>

All cards of a structural artifact should be in lexicographical order.
The Fossil implementation verifies this and rejects any structural
artifact which has out-of-order cards.  Futhermore, when Fossil is
generating new structural artifacts, it runs the generated artifact
through the parser to confirm that all cards really are in the correct
order before committing the transaction.  In this way, Fossil prevents
bugs in the code from accidentally inserting misformatted artifacts.
The test parse of newly created artifacts is part of the
[./selfcheck.wiki|self-check strategy] of Fossil.  It takes a
few more CPU cycles to double check each artifact before inserting it.
The developers consider those CPU cycles well-spent.

However, the card-order safety check was accidentally disabled due to
[15d04de574383d61|a bug].
And while that bug was lurking undetected in the code,
[5e67a7f4041a36ad|another bug] caused the N cards of Technical Notes
to occur after the P card rather than before.
Thus for a span of several years, Technical Note artifacts were being
inserted into Fossil repositories that had their N and P cards in the
wrong order.

Both bugs have now been fixed.  However, to prevent historical
Technical Note artifacts that were inserted by users in good faith
from being rejected by newer Fossil builds, the card ordering
requirement is relaxed slightly.  The actual implementation is this:

<blockquote>
"All cards must be in strict lexicographic order, except that the
N and P cards of a Technical Note artifact are allowed to be
interchanged."
</blockquote>

Future versions of Fossil might strengthen this slightly to only allow
the out of order N and P cards for Technical Notes entered before
a certain date.

<h3>4.2 R-Card Hash Calculation</h3>

Given a manifest file named <tt>MF</tt>, the following Bash shell code
demonstrates how to compute the value of the <b>R</b> card in that manifest.
This example uses manifest [28987096ac]. Lines starting with <tt>#</tt> are
shell input and other lines are output. This demonstration assumes that the
file versions represented by the input manifest are checked out
under the current directory.