Fossil

Check-in [ee710cc171]
Login

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | clone-resume
Files: files | file ages | folders
SHA3-256: ee710cc171e6164bfad85e6b38f63f7868e3d2706236b40ad078ae898cac5962
User & Date: andybradford 2023-11-24 21:39:44.356
Original Comment: Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card. a
Context
2023-11-24
22:14
Need to reset the server-code so that the user password is correct. Might as well also reset the project-code. check-in: 61e0ced9bf user: andybradford tags: clone-resume
21:39
Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card. check-in: ee710cc171 user: andybradford tags: clone-resume
2023-11-20
19:16
On the activity report page, make sure the graph bar with the expected changes for the current week/month is not displayed on a separate line. check-in: 698dc59ec1 user: florian tags: trunk
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/clone.c.
127
128
129
130
131
132
133

134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

154
155
156
157
158
159
160
**    -B|--httpauth USER:PASS    Add HTTP Basic Authorization to requests
**    --nested                   Allow opening a repository inside an opened
**                               check-out
**    --nocompress               Omit extra delta compression
**    --no-open                  Clone only.  Do not open a check-out.
**    --once                     Don't remember the URI.
**    --private                  Also clone private branches

**    --save-http-password       Remember the HTTP password without asking
**    -c|--ssh-command SSH       Use SSH as the "ssh" command
**    --ssl-identity FILENAME    Use the SSL identity if requested by the server
**    --transport-command CMD    Use CMD to move messages to the server and back
**    -u|--unversioned           Also sync unversioned content
**    -v|--verbose               Show more statistics in output
**    --workdir DIR              Also open a check-out in DIR
**
** See also: [[init]], [[open]]
*/
void clone_cmd(void){
  char *zPassword;
  const char *zDefaultUser;   /* Optional name of the default user */
  const char *zHttpAuth;      /* HTTP Authorization user:pass information */
  int nErr = 0;
  int urlFlags = URL_PROMPT_PW | URL_REMEMBER;
  int syncFlags = SYNC_CLONE;
  int noCompress = find_option("nocompress",0,0)!=0;
  int noOpen = find_option("no-open",0,0)!=0;
  int allowNested = find_option("nested",0,0)!=0; /* Used by open */

  const char *zRepo = 0;      /* Name of the new local repository file */
  const char *zWorkDir = 0;   /* Open in this directory, if not zero */


  /* Also clone private branches */
  if( find_option("private",0,0)!=0 ) syncFlags |= SYNC_PRIVATE;
  if( find_option("once",0,0)!=0) urlFlags &= ~URL_REMEMBER;







>




















>







127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
**    -B|--httpauth USER:PASS    Add HTTP Basic Authorization to requests
**    --nested                   Allow opening a repository inside an opened
**                               check-out
**    --nocompress               Omit extra delta compression
**    --no-open                  Clone only.  Do not open a check-out.
**    --once                     Don't remember the URI.
**    --private                  Also clone private branches
**    --resume                   Resume a failed clone
**    --save-http-password       Remember the HTTP password without asking
**    -c|--ssh-command SSH       Use SSH as the "ssh" command
**    --ssl-identity FILENAME    Use the SSL identity if requested by the server
**    --transport-command CMD    Use CMD to move messages to the server and back
**    -u|--unversioned           Also sync unversioned content
**    -v|--verbose               Show more statistics in output
**    --workdir DIR              Also open a check-out in DIR
**
** See also: [[init]], [[open]]
*/
void clone_cmd(void){
  char *zPassword;
  const char *zDefaultUser;   /* Optional name of the default user */
  const char *zHttpAuth;      /* HTTP Authorization user:pass information */
  int nErr = 0;
  int urlFlags = URL_PROMPT_PW | URL_REMEMBER;
  int syncFlags = SYNC_CLONE;
  int noCompress = find_option("nocompress",0,0)!=0;
  int noOpen = find_option("no-open",0,0)!=0;
  int allowNested = find_option("nested",0,0)!=0; /* Used by open */
  int bResume = find_option("resume",0,0)!=0; /* Resume a failed clone */
  const char *zRepo = 0;      /* Name of the new local repository file */
  const char *zWorkDir = 0;   /* Open in this directory, if not zero */


  /* Also clone private branches */
  if( find_option("private",0,0)!=0 ) syncFlags |= SYNC_PRIVATE;
  if( find_option("once",0,0)!=0) urlFlags &= ~URL_REMEMBER;
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
    }
    zRepo = mprintf("./%s.fossil", zBase);
    if( zWorkDir==0 ){
      zWorkDir = mprintf("./%s", zBase);
    }
    fossil_free(zBase);
  }  
  if( -1 != file_size(zRepo, ExtFILE) ){
    fossil_fatal("file already exists: %s", zRepo);
  }
  /* Fail before clone if open will fail because inside an open check-out */
  if( zWorkDir!=0 && zWorkDir[0]!=0 && !noOpen ){
    if( db_open_local_v2(0, allowNested) ){
      fossil_fatal("there is already an open tree at %s", g.zLocalRoot);
    }







|







197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
    }
    zRepo = mprintf("./%s.fossil", zBase);
    if( zWorkDir==0 ){
      zWorkDir = mprintf("./%s", zBase);
    }
    fossil_free(zBase);
  }  
  if( -1 != file_size(zRepo, ExtFILE) && bResume==0 ){
    fossil_fatal("file already exists: %s", zRepo);
  }
  /* Fail before clone if open will fail because inside an open check-out */
  if( zWorkDir!=0 && zWorkDir[0]!=0 && !noOpen ){
    if( db_open_local_v2(0, allowNested) ){
      fossil_fatal("there is already an open tree at %s", g.zLocalRoot);
    }
224
225
226
227
228
229
230






231
232
233
234
235
236
237
238
239


240
241
242
243
244
245
246
      g.zLogin = zDefaultUser;
    }else{
      g.zLogin = db_text(0, "SELECT login FROM user WHERE cap LIKE '%%s%%'");
    }
    fossil_print("Repository cloned into %s\n", zRepo);
  }else{
    db_close_config();






    db_create_repository(zRepo);
    db_open_repository(zRepo);
    db_open_config(0,0);
    db_begin_transaction();
    db_record_repository_filename(zRepo);
    db_initial_setup(0, 0, zDefaultUser);
    user_select();
    db_set("content-schema", CONTENT_SCHEMA, 0);
    db_set("aux-schema", AUX_SCHEMA_MAX, 0);


    db_set("rebuilt", get_version(), 0);
    db_unset("hash-policy", 0);
    remember_or_get_http_auth(zHttpAuth, urlFlags & URL_REMEMBER, g.argv[2]);
    url_remember();
    if( g.zSSLIdentity!=0 ){
      /* If the --ssl-identity option was specified, store it as a setting */
      Blob fn;







>
>
>
>
>
>
|
|
|
|
|
|
|
|
|
>
>







226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
      g.zLogin = zDefaultUser;
    }else{
      g.zLogin = db_text(0, "SELECT login FROM user WHERE cap LIKE '%%s%%'");
    }
    fossil_print("Repository cloned into %s\n", zRepo);
  }else{
    db_close_config();
    if( bResume ){
      db_open_repository(zRepo);
      db_open_config(0,0);
      db_begin_transaction();
      user_select();
    }else{
      db_create_repository(zRepo);
      db_open_repository(zRepo);
      db_open_config(0,0);
      db_begin_transaction();
      db_record_repository_filename(zRepo);
      db_initial_setup(0, 0, zDefaultUser);
      user_select();
      db_set("content-schema", CONTENT_SCHEMA, 0);
      db_set("aux-schema", AUX_SCHEMA_MAX, 0);
      db_set_int("aux-clone-seqno", 1, 0);
    }
    db_set("rebuilt", get_version(), 0);
    db_unset("hash-policy", 0);
    remember_or_get_http_auth(zHttpAuth, urlFlags & URL_REMEMBER, g.argv[2]);
    url_remember();
    if( g.zSSLIdentity!=0 ){
      /* If the --ssl-identity option was specified, store it as a setting */
      Blob fn;
261
262
263
264
265
266
267
268
269
270
271
272




273


274
275
276
277
278
279

280
281
282
283
284
285
286
    url_enable_proxy(0);
    clone_ssh_db_set_options();
    url_get_password_if_needed();
    g.xlinkClusterOnly = 1;
    nErr = client_sync(syncFlags,CONFIGSET_ALL,0,0);
    g.xlinkClusterOnly = 0;
    verify_cancel();
    db_end_transaction(0);
    db_close(1);
    if( nErr ){
      file_delete(zRepo);
      fossil_fatal("server returned an error - clone aborted");




    }


    db_open_repository(zRepo);
  }
  db_begin_transaction();
  if( db_exists("SELECT 1 FROM delta WHERE srcId IN phantom") ){
    fossil_fatal("there are unresolved deltas -"
                 " the clone is probably incomplete and unusable.");

  }
  fossil_print("Rebuilding repository meta-data...\n");
  rebuild_db(1, 0);
  if( !noCompress ){
    int nDelta = 0;
    i64 nByte;
    fossil_print("Extra delta compression... "); fflush(stdout);







<
<

<
|
>
>
>
>

>
>




|
|
>







271
272
273
274
275
276
277


278

279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
    url_enable_proxy(0);
    clone_ssh_db_set_options();
    url_get_password_if_needed();
    g.xlinkClusterOnly = 1;
    nErr = client_sync(syncFlags,CONFIGSET_ALL,0,0);
    g.xlinkClusterOnly = 0;
    verify_cancel();


    if( nErr ){

      fossil_warning("server returned an error - clone incomplete");
    }else{
      db_unprotect(PROTECT_CONFIG);
      db_multi_exec("DELETE FROM config WHERE name = 'aux-clone-seqno';");
      db_protect_pop();
    }
    db_end_transaction(0);
    db_close(1);
    db_open_repository(zRepo);
  }
  db_begin_transaction();
  if( db_exists("SELECT 1 FROM delta WHERE srcId IN phantom") ){
    fossil_warning("there are unresolved deltas -"
                 " the clone is probably incomplete and unusable.\n"
                 "It may be possible to continue clone with --resume.");
  }
  fossil_print("Rebuilding repository meta-data...\n");
  rebuild_db(1, 0);
  if( !noCompress ){
    int nDelta = 0;
    i64 nByte;
    fossil_print("Extra delta compression... "); fflush(stdout);
Changes to src/db.c.
4235
4236
4237
4238
4239
4240
4241



4242
4243
4244
4245
4246
4247
4248
    fossil_fatal("the --repodir option only makes sense if the REPOSITORY "
                 "argument is a URI that begins with http:, https:, ssh:, "
                 "or file:");
  }

  db_open_config(0,0);
  db_open_repository(zRepo);




  /* Figure out which revision to open. */
  if( !emptyFlag ){
    if( g.argc==4 ){
      g.zOpenRevision = g.argv[3];
    }else if( db_exists("SELECT 1 FROM event WHERE type='ci'") ){
      g.zOpenRevision = db_get("main-branch", 0);







>
>
>







4235
4236
4237
4238
4239
4240
4241
4242
4243
4244
4245
4246
4247
4248
4249
4250
4251
    fossil_fatal("the --repodir option only makes sense if the REPOSITORY "
                 "argument is a URI that begins with http:, https:, ssh:, "
                 "or file:");
  }

  db_open_config(0,0);
  db_open_repository(zRepo);
  if( db_get_int("aux-clone-seqno",0)!=0 ){
    fossil_fatal("This repository appears to be an incomplete clone.");
  }

  /* Figure out which revision to open. */
  if( !emptyFlag ){
    if( g.argc==4 ){
      g.zOpenRevision = g.argv[3];
    }else if( db_exists("SELECT 1 FROM event WHERE type='ci'") ){
      g.zOpenRevision = db_get("main-branch", 0);
Changes to src/xfer.c.
1463
1464
1465
1466
1467
1468
1469




1470
1471
1472
1473
1474
1475
1476
          cgi_set_content_type("application/x-fossil-uncompressed");
        }
        blob_is_int(&xfer.aToken[2], &seqno);
        max = db_int(0, "SELECT max(rid) FROM blob");
        while( xfer.mxSend>(int)blob_size(xfer.pOut) && seqno<=max){
          if( time(NULL) >= xfer.maxTime ) break;
          if( iVers>=3 ){




            send_compressed_file(&xfer, seqno);
          }else{
            send_file(&xfer, seqno, 0, 1);
          }
          seqno++;
        }
        if( seqno>max ) seqno = 0;







>
>
>
>







1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
          cgi_set_content_type("application/x-fossil-uncompressed");
        }
        blob_is_int(&xfer.aToken[2], &seqno);
        max = db_int(0, "SELECT max(rid) FROM blob");
        while( xfer.mxSend>(int)blob_size(xfer.pOut) && seqno<=max){
          if( time(NULL) >= xfer.maxTime ) break;
          if( iVers>=3 ){
            if( seqno==1 ){
              send_all(&xfer);
              if( xfer.syncPrivate ) send_private(&xfer);
            }
            send_compressed_file(&xfer, seqno);
          }else{
            send_file(&xfer, seqno, 0, 1);
          }
          seqno++;
        }
        if( seqno>max ) seqno = 0;
1994
1995
1996
1997
1998
1999
2000






2001
2002
2003
2004
2005
2006
2007
  if( db_get_boolean("dont-push", 0) ) syncFlags &= ~SYNC_PUSH;
  if( (syncFlags & (SYNC_PUSH|SYNC_PULL|SYNC_CLONE|SYNC_UNVERSIONED))==0
     && configRcvMask==0
     && configSendMask==0
  ){
    return 0;  /* Nothing to do */
  }







  /* Compute an appropriate project code.  zPCode is the project code
  ** for the local repository.  zAltPCode will usually be NULL, but might
  ** also be an alternative project code to expect on the server.  When
  ** zAltPCode is not NULL, that means we are doing a cross-project import -
  ** in other words, reading content from one project into a different
  ** project.







>
>
>
>
>
>







1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
  if( db_get_boolean("dont-push", 0) ) syncFlags &= ~SYNC_PUSH;
  if( (syncFlags & (SYNC_PUSH|SYNC_PULL|SYNC_CLONE|SYNC_UNVERSIONED))==0
     && configRcvMask==0
     && configSendMask==0
  ){
    return 0;  /* Nothing to do */
  }

  if( (syncFlags & SYNC_CLONE)==0 && db_get_int("aux-clone-seqno",0) ){
    fossil_fatal("Unable to synchronize due to incomplete clone.");
  }else{
    cloneSeqno = db_get_int("aux-clone-seqno",1);
  }

  /* Compute an appropriate project code.  zPCode is the project code
  ** for the local repository.  zAltPCode will usually be NULL, but might
  ** also be an alternative project code to expect on the server.  When
  ** zAltPCode is not NULL, that means we are doing a cross-project import -
  ** in other words, reading content from one project into a different
  ** project.
2808
2809
2810
2811
2812
2813
2814





2815
2816
2817
2818
2819
2820
2821
      fossil_print(zBriefFormat /*works-like:"%d%d%d"*/,
                   nRoundtrip, nArtifactSent, nArtifactRcvd);
    }
    nUncRcvd += blob_size(&recv);
    blob_reset(&recv);
    nCycle++;






    /* Set go to 1 if we need to continue the sync/push/pull/clone for
    ** another round.  Set go to 0 if it is time to quit. */
    nFileRecv = xfer.nFileRcvd + xfer.nDeltaRcvd + xfer.nDanglingFile;
    if( (nFileRecv>0 || newPhantom) && db_exists("SELECT 1 FROM phantom") ){
      go = 1;
      mxPhantomReq = nFileRecv*2;
      if( mxPhantomReq<200 ) mxPhantomReq = 200;







>
>
>
>
>







2818
2819
2820
2821
2822
2823
2824
2825
2826
2827
2828
2829
2830
2831
2832
2833
2834
2835
2836
      fossil_print(zBriefFormat /*works-like:"%d%d%d"*/,
                   nRoundtrip, nArtifactSent, nArtifactRcvd);
    }
    nUncRcvd += blob_size(&recv);
    blob_reset(&recv);
    nCycle++;

    /* Record the current cloneSeqno in the event that clone fails to enable
    ** the ability to resume from this same point in clone.  */
    if( (syncFlags & SYNC_CLONE)!=0 ){
      db_set_int("aux-clone-seqno", cloneSeqno, 0);
    }
    /* Set go to 1 if we need to continue the sync/push/pull/clone for
    ** another round.  Set go to 0 if it is time to quit. */
    nFileRecv = xfer.nFileRcvd + xfer.nDeltaRcvd + xfer.nDanglingFile;
    if( (nFileRecv>0 || newPhantom) && db_exists("SELECT 1 FROM phantom") ){
      go = 1;
      mxPhantomReq = nFileRecv*2;
      if( mxPhantomReq<200 ) mxPhantomReq = 200;