Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
Overview
| Comment: | Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card. |
|---|---|
| Downloads: | Tarball | ZIP archive |
| Timelines: | family | ancestors | descendants | both | clone-resume |
| Files: | files | file ages | folders |
| SHA3-256: |
ee710cc171e6164bfad85e6b38f63f78 |
| User & Date: | andybradford 2023-11-24 21:39:44.356 |
| Original Comment: | Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card. a |
Context
|
2023-11-24
| ||
| 22:14 | Need to reset the server-code so that the user password is correct. Might as well also reset the project-code. check-in: 61e0ced9bf user: andybradford tags: clone-resume | |
| 21:39 | Implement the ability to resume a clone that has failed. This is a variation on the attempt made in [ec26471439] that was never completed. The significant difference in the use of the "clone protocol" which uses cfile cards to complete the synchronization rather than the "sync protocol" which takes much longer using the file card. check-in: ee710cc171 user: andybradford tags: clone-resume | |
|
2023-11-20
| ||
| 19:16 | On the activity report page, make sure the graph bar with the expected changes for the current week/month is not displayed on a separate line. check-in: 698dc59ec1 user: florian tags: trunk | |
Changes
Changes to src/clone.c.
| ︙ | ︙ | |||
127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
** -B|--httpauth USER:PASS Add HTTP Basic Authorization to requests
** --nested Allow opening a repository inside an opened
** check-out
** --nocompress Omit extra delta compression
** --no-open Clone only. Do not open a check-out.
** --once Don't remember the URI.
** --private Also clone private branches
** --save-http-password Remember the HTTP password without asking
** -c|--ssh-command SSH Use SSH as the "ssh" command
** --ssl-identity FILENAME Use the SSL identity if requested by the server
** --transport-command CMD Use CMD to move messages to the server and back
** -u|--unversioned Also sync unversioned content
** -v|--verbose Show more statistics in output
** --workdir DIR Also open a check-out in DIR
**
** See also: [[init]], [[open]]
*/
void clone_cmd(void){
char *zPassword;
const char *zDefaultUser; /* Optional name of the default user */
const char *zHttpAuth; /* HTTP Authorization user:pass information */
int nErr = 0;
int urlFlags = URL_PROMPT_PW | URL_REMEMBER;
int syncFlags = SYNC_CLONE;
int noCompress = find_option("nocompress",0,0)!=0;
int noOpen = find_option("no-open",0,0)!=0;
int allowNested = find_option("nested",0,0)!=0; /* Used by open */
const char *zRepo = 0; /* Name of the new local repository file */
const char *zWorkDir = 0; /* Open in this directory, if not zero */
/* Also clone private branches */
if( find_option("private",0,0)!=0 ) syncFlags |= SYNC_PRIVATE;
if( find_option("once",0,0)!=0) urlFlags &= ~URL_REMEMBER;
| > > | 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 |
** -B|--httpauth USER:PASS Add HTTP Basic Authorization to requests
** --nested Allow opening a repository inside an opened
** check-out
** --nocompress Omit extra delta compression
** --no-open Clone only. Do not open a check-out.
** --once Don't remember the URI.
** --private Also clone private branches
** --resume Resume a failed clone
** --save-http-password Remember the HTTP password without asking
** -c|--ssh-command SSH Use SSH as the "ssh" command
** --ssl-identity FILENAME Use the SSL identity if requested by the server
** --transport-command CMD Use CMD to move messages to the server and back
** -u|--unversioned Also sync unversioned content
** -v|--verbose Show more statistics in output
** --workdir DIR Also open a check-out in DIR
**
** See also: [[init]], [[open]]
*/
void clone_cmd(void){
char *zPassword;
const char *zDefaultUser; /* Optional name of the default user */
const char *zHttpAuth; /* HTTP Authorization user:pass information */
int nErr = 0;
int urlFlags = URL_PROMPT_PW | URL_REMEMBER;
int syncFlags = SYNC_CLONE;
int noCompress = find_option("nocompress",0,0)!=0;
int noOpen = find_option("no-open",0,0)!=0;
int allowNested = find_option("nested",0,0)!=0; /* Used by open */
int bResume = find_option("resume",0,0)!=0; /* Resume a failed clone */
const char *zRepo = 0; /* Name of the new local repository file */
const char *zWorkDir = 0; /* Open in this directory, if not zero */
/* Also clone private branches */
if( find_option("private",0,0)!=0 ) syncFlags |= SYNC_PRIVATE;
if( find_option("once",0,0)!=0) urlFlags &= ~URL_REMEMBER;
|
| ︙ | ︙ | |||
195 196 197 198 199 200 201 |
}
zRepo = mprintf("./%s.fossil", zBase);
if( zWorkDir==0 ){
zWorkDir = mprintf("./%s", zBase);
}
fossil_free(zBase);
}
| | | 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 |
}
zRepo = mprintf("./%s.fossil", zBase);
if( zWorkDir==0 ){
zWorkDir = mprintf("./%s", zBase);
}
fossil_free(zBase);
}
if( -1 != file_size(zRepo, ExtFILE) && bResume==0 ){
fossil_fatal("file already exists: %s", zRepo);
}
/* Fail before clone if open will fail because inside an open check-out */
if( zWorkDir!=0 && zWorkDir[0]!=0 && !noOpen ){
if( db_open_local_v2(0, allowNested) ){
fossil_fatal("there is already an open tree at %s", g.zLocalRoot);
}
|
| ︙ | ︙ | |||
224 225 226 227 228 229 230 |
g.zLogin = zDefaultUser;
}else{
g.zLogin = db_text(0, "SELECT login FROM user WHERE cap LIKE '%%s%%'");
}
fossil_print("Repository cloned into %s\n", zRepo);
}else{
db_close_config();
| > > > > > > | | | | | | | | | > > | 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 |
g.zLogin = zDefaultUser;
}else{
g.zLogin = db_text(0, "SELECT login FROM user WHERE cap LIKE '%%s%%'");
}
fossil_print("Repository cloned into %s\n", zRepo);
}else{
db_close_config();
if( bResume ){
db_open_repository(zRepo);
db_open_config(0,0);
db_begin_transaction();
user_select();
}else{
db_create_repository(zRepo);
db_open_repository(zRepo);
db_open_config(0,0);
db_begin_transaction();
db_record_repository_filename(zRepo);
db_initial_setup(0, 0, zDefaultUser);
user_select();
db_set("content-schema", CONTENT_SCHEMA, 0);
db_set("aux-schema", AUX_SCHEMA_MAX, 0);
db_set_int("aux-clone-seqno", 1, 0);
}
db_set("rebuilt", get_version(), 0);
db_unset("hash-policy", 0);
remember_or_get_http_auth(zHttpAuth, urlFlags & URL_REMEMBER, g.argv[2]);
url_remember();
if( g.zSSLIdentity!=0 ){
/* If the --ssl-identity option was specified, store it as a setting */
Blob fn;
|
| ︙ | ︙ | |||
261 262 263 264 265 266 267 |
url_enable_proxy(0);
clone_ssh_db_set_options();
url_get_password_if_needed();
g.xlinkClusterOnly = 1;
nErr = client_sync(syncFlags,CONFIGSET_ALL,0,0);
g.xlinkClusterOnly = 0;
verify_cancel();
| < < < | > > > > > > | | > | 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 |
url_enable_proxy(0);
clone_ssh_db_set_options();
url_get_password_if_needed();
g.xlinkClusterOnly = 1;
nErr = client_sync(syncFlags,CONFIGSET_ALL,0,0);
g.xlinkClusterOnly = 0;
verify_cancel();
if( nErr ){
fossil_warning("server returned an error - clone incomplete");
}else{
db_unprotect(PROTECT_CONFIG);
db_multi_exec("DELETE FROM config WHERE name = 'aux-clone-seqno';");
db_protect_pop();
}
db_end_transaction(0);
db_close(1);
db_open_repository(zRepo);
}
db_begin_transaction();
if( db_exists("SELECT 1 FROM delta WHERE srcId IN phantom") ){
fossil_warning("there are unresolved deltas -"
" the clone is probably incomplete and unusable.\n"
"It may be possible to continue clone with --resume.");
}
fossil_print("Rebuilding repository meta-data...\n");
rebuild_db(1, 0);
if( !noCompress ){
int nDelta = 0;
i64 nByte;
fossil_print("Extra delta compression... "); fflush(stdout);
|
| ︙ | ︙ |
Changes to src/db.c.
| ︙ | ︙ | |||
4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 |
fossil_fatal("the --repodir option only makes sense if the REPOSITORY "
"argument is a URI that begins with http:, https:, ssh:, "
"or file:");
}
db_open_config(0,0);
db_open_repository(zRepo);
/* Figure out which revision to open. */
if( !emptyFlag ){
if( g.argc==4 ){
g.zOpenRevision = g.argv[3];
}else if( db_exists("SELECT 1 FROM event WHERE type='ci'") ){
g.zOpenRevision = db_get("main-branch", 0);
| > > > | 4235 4236 4237 4238 4239 4240 4241 4242 4243 4244 4245 4246 4247 4248 4249 4250 4251 |
fossil_fatal("the --repodir option only makes sense if the REPOSITORY "
"argument is a URI that begins with http:, https:, ssh:, "
"or file:");
}
db_open_config(0,0);
db_open_repository(zRepo);
if( db_get_int("aux-clone-seqno",0)!=0 ){
fossil_fatal("This repository appears to be an incomplete clone.");
}
/* Figure out which revision to open. */
if( !emptyFlag ){
if( g.argc==4 ){
g.zOpenRevision = g.argv[3];
}else if( db_exists("SELECT 1 FROM event WHERE type='ci'") ){
g.zOpenRevision = db_get("main-branch", 0);
|
| ︙ | ︙ |
Changes to src/xfer.c.
| ︙ | ︙ | |||
1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 |
cgi_set_content_type("application/x-fossil-uncompressed");
}
blob_is_int(&xfer.aToken[2], &seqno);
max = db_int(0, "SELECT max(rid) FROM blob");
while( xfer.mxSend>(int)blob_size(xfer.pOut) && seqno<=max){
if( time(NULL) >= xfer.maxTime ) break;
if( iVers>=3 ){
send_compressed_file(&xfer, seqno);
}else{
send_file(&xfer, seqno, 0, 1);
}
seqno++;
}
if( seqno>max ) seqno = 0;
| > > > > | 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 |
cgi_set_content_type("application/x-fossil-uncompressed");
}
blob_is_int(&xfer.aToken[2], &seqno);
max = db_int(0, "SELECT max(rid) FROM blob");
while( xfer.mxSend>(int)blob_size(xfer.pOut) && seqno<=max){
if( time(NULL) >= xfer.maxTime ) break;
if( iVers>=3 ){
if( seqno==1 ){
send_all(&xfer);
if( xfer.syncPrivate ) send_private(&xfer);
}
send_compressed_file(&xfer, seqno);
}else{
send_file(&xfer, seqno, 0, 1);
}
seqno++;
}
if( seqno>max ) seqno = 0;
|
| ︙ | ︙ | |||
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 |
if( db_get_boolean("dont-push", 0) ) syncFlags &= ~SYNC_PUSH;
if( (syncFlags & (SYNC_PUSH|SYNC_PULL|SYNC_CLONE|SYNC_UNVERSIONED))==0
&& configRcvMask==0
&& configSendMask==0
){
return 0; /* Nothing to do */
}
/* Compute an appropriate project code. zPCode is the project code
** for the local repository. zAltPCode will usually be NULL, but might
** also be an alternative project code to expect on the server. When
** zAltPCode is not NULL, that means we are doing a cross-project import -
** in other words, reading content from one project into a different
** project.
| > > > > > > | 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 |
if( db_get_boolean("dont-push", 0) ) syncFlags &= ~SYNC_PUSH;
if( (syncFlags & (SYNC_PUSH|SYNC_PULL|SYNC_CLONE|SYNC_UNVERSIONED))==0
&& configRcvMask==0
&& configSendMask==0
){
return 0; /* Nothing to do */
}
if( (syncFlags & SYNC_CLONE)==0 && db_get_int("aux-clone-seqno",0) ){
fossil_fatal("Unable to synchronize due to incomplete clone.");
}else{
cloneSeqno = db_get_int("aux-clone-seqno",1);
}
/* Compute an appropriate project code. zPCode is the project code
** for the local repository. zAltPCode will usually be NULL, but might
** also be an alternative project code to expect on the server. When
** zAltPCode is not NULL, that means we are doing a cross-project import -
** in other words, reading content from one project into a different
** project.
|
| ︙ | ︙ | |||
2808 2809 2810 2811 2812 2813 2814 2815 2816 2817 2818 2819 2820 2821 |
fossil_print(zBriefFormat /*works-like:"%d%d%d"*/,
nRoundtrip, nArtifactSent, nArtifactRcvd);
}
nUncRcvd += blob_size(&recv);
blob_reset(&recv);
nCycle++;
/* Set go to 1 if we need to continue the sync/push/pull/clone for
** another round. Set go to 0 if it is time to quit. */
nFileRecv = xfer.nFileRcvd + xfer.nDeltaRcvd + xfer.nDanglingFile;
if( (nFileRecv>0 || newPhantom) && db_exists("SELECT 1 FROM phantom") ){
go = 1;
mxPhantomReq = nFileRecv*2;
if( mxPhantomReq<200 ) mxPhantomReq = 200;
| > > > > > | 2818 2819 2820 2821 2822 2823 2824 2825 2826 2827 2828 2829 2830 2831 2832 2833 2834 2835 2836 |
fossil_print(zBriefFormat /*works-like:"%d%d%d"*/,
nRoundtrip, nArtifactSent, nArtifactRcvd);
}
nUncRcvd += blob_size(&recv);
blob_reset(&recv);
nCycle++;
/* Record the current cloneSeqno in the event that clone fails to enable
** the ability to resume from this same point in clone. */
if( (syncFlags & SYNC_CLONE)!=0 ){
db_set_int("aux-clone-seqno", cloneSeqno, 0);
}
/* Set go to 1 if we need to continue the sync/push/pull/clone for
** another round. Set go to 0 if it is time to quit. */
nFileRecv = xfer.nFileRcvd + xfer.nDeltaRcvd + xfer.nDanglingFile;
if( (nFileRecv>0 || newPhantom) && db_exists("SELECT 1 FROM phantom") ){
go = 1;
mxPhantomReq = nFileRecv*2;
if( mxPhantomReq<200 ) mxPhantomReq = 200;
|
| ︙ | ︙ |