$<project_name>: $<title>

Difference From:

To:

Differences From:

To:

Artifact %s(zUuid):

Artifact %s(zUuid)

Changes to ticket %s(zTktName)

Changes to ticket + @ %s(zTktName)

Changes to ticket %s(zTktName)

Tags And Properties

What Is An "Event"?

Viewing Events

Creating And Editing Events

1.0 The Manifest

5.0 Ticket Changes

6.0 Attachments

7.0 Events

7.0 Card Summary

8.0 Card Summary

Web Interface Authentication

Various security concerns with hosted repositories

Links For Fossil Users:

$
$</div> + <div class="status"><nobr><th1> + if {[info exists login]} { + puts "Logged in as $login" + } else { + puts "Not logged in" + } + </th1></nobr></div> +</div> +<div class="mainmenu"><th1> +html "<a href=''$baseurl$index_page''>Home</a> " +if {[anycap jor]} { + html "<a href=''$baseurl/timeline''>Timeline</a> " +} +if {[hascap oh]} { + html "<a href=''$baseurl/dir?ci=tip''>Files</a> " +} +if {[hascap o]} { + html "<a href=''$baseurl/leaves''>Leaves</a> " + html "<a href=''$baseurl/brlist''>Branches</a> " + html "<a href=''$baseurl/taglist''>Tags</a> " +} +if {[hascap r]} { + html "<a href=''$baseurl/reportlist''>Tickets</a> " +} +if {[hascap j]} { + html "<a href=''$baseurl/wiki''>Wiki</a> " +} +if {[hascap s]} { + html "<a href=''$baseurl/setup''>Admin</a> " +} elseif {[hascap a]} { + html "<a href=''$baseurl/setup_ulist''>Users</a> " +} +if {[info exists login]} { + html "<a href=''$baseurl/login''>Logout</a> " +} else { + html "<a href=''$baseurl/login''>Login</a> " +} +</th1></div> ADDED skins/default1/info.txt Index: skins/default1/info.txt ================================================================== --- skins/default1/info.txt +++ skins/default1/info.txt @@ -0,0 +1,5 @@ +Title: Plain Gray +Description: A black-and-white theme with the project title in a bar across the + top and no logo image. +Author: Unknown. + ADDED skins/default1/style.css Index: skins/default1/style.css ================================================================== --- skins/default1/style.css +++ skins/default1/style.css @@ -0,0 +1,125 @@ +/* General settings for the entire page */ +body { + margin: 0ex 1ex; + padding: 0px; + background-color: white; + font-family: sans-serif; +} + +/* The project logo in the upper left-hand corner of each page */ +div.logo { + display: table-row; + text-align: center; + /* vertical-align: bottom;*/ + font-size: 2em; + font-weight: bold; + background-color: #707070; + color: #ffffff; + min-width: 200px; +} + +/* The page title centered at the top of each page */ +div.title { + display: table-cell; + font-size: 1.5em; + font-weight: bold; + text-align: center; + padding: 0 0 0 10px; + color: #404040; + vertical-align: bottom; + width: 100%; +} + +/* The login status message in the top right-hand corner */ +div.status { + display: table-cell; + text-align: right; + vertical-align: bottom; + color: #404040; + font-size: 0.8em; + font-weight: bold; + min-width: 200px; +} + +/* The header across the top of the page */ +div.header { + display: table; + width: 100%; +} + +/* The main menu bar that appears at the top of the page beneath the header */ +div.mainmenu { + padding: 5px 10px 5px 10px; + font-size: 0.9em; + font-weight: bold; + text-align: center; + letter-spacing: 1px; + background-color: #404040; + color: white; +} + +/* The submenu bar that *sometimes* appears below the main menu */ +div.submenu { + padding: 3px 10px 3px 0px; + font-size: 0.9em; + text-align: center; + background-color: #606060; + color: white; +} +div.mainmenu a, div.mainmenu a:visited, div.submenu a, div.submenu a:visited { + padding: 3px 10px 3px 10px; + color: white; + text-decoration: none; +} +div.mainmenu a:hover, div.submenu a:hover { + color: #404040; + background-color: white; +} + +/* All page content from the bottom of the menu or submenu down to the footer */ +div.content { + padding: 0ex 0ex 0ex 0ex; +} +/* Hyperlink colors */ +div.content a { color: #604000; } +div.content a:link { color: #604000;} +div.content a:visited { color: #600000; } + +/* Some pages have section dividers */ +div.section { + margin-bottom: 0px; + margin-top: 1em; + padding: 1px 1px 1px 1px; + font-size: 1.2em; + font-weight: bold; + background-color: #404040; + color: white; +} + +/* The "Date" that occurs on the left hand side of timelines */ +div.divider { + background: #a0a0a0; + border: 2px #505050 solid; + font-size: 1em; font-weight: normal; + padding: .25em; + margin: .2em 0 .2em 0; + float: left; + clear: left; +} + +/* The footer at the very bottom of the page */ +div.footer { + font-size: 0.8em; + margin-top: 12px; + padding: 5px 10px 5px 10px; + text-align: right; + background-color: #404040; + color: white; +} + +/* The label/value pairs on (for example) the vinfo page */ +table.label-value th { + vertical-align: top; + text-align: right; + padding: 0.2ex 2ex; +} ADDED skins/default2/footer.html Index: skins/default2/footer.html ================================================================== --- skins/default2/footer.html +++ skins/default2/footer.html @@ -0,0 +1,4 @@ +<div class="footer"> +Fossil version $manifest_version $manifest_date +</div> +</body></html> ADDED skins/default2/header.html Index: skins/default2/header.html ================================================================== --- skins/default2/header.html +++ skins/default2/header.html @@ -0,0 +1,51 @@ +<html> +<head> +<title>$<project_name>: $<title> + + + + +

$</div> + <div class="status"> + <div class="logo"><nobr>$<project_name></nobr></div><br/> + <nobr><th1> + if {[info exists login]} { + puts "Logged in as $login" + } else { + puts "Not logged in" + } + </th1></nobr></div> +</div> +<div class="mainmenu"><th1> +html "<a href=''$baseurl$index_page''>Home</a> " +if {[anycap jor]} { + html "<a href=''$baseurl/timeline''>Timeline</a> " +} +if {[hascap oh]} { + html "<a href=''$baseurl/dir?ci=tip''>Files</a> " +} +if {[hascap o]} { + html "<a href=''$baseurl/leaves''>Leaves</a> " + html "<a href=''$baseurl/brlist''>Branches</a> " + html "<a href=''$baseurl/taglist''>Tags</a> " +} +if {[hascap r]} { + html "<a href=''$baseurl/reportlist''>Tickets</a> " +} +if {[hascap j]} { + html "<a href=''$baseurl/wiki''>Wiki</a> " +} +if {[hascap s]} { + html "<a href=''$baseurl/setup''>Admin</a> " +} elseif {[hascap a]} { + html "<a href=''$baseurl/setup_ulist''>Users</a> " +} +if {[info exists login]} { + html "<a href=''$baseurl/login''>Logout</a> " +} else { + html "<a href=''$baseurl/login''>Login</a> " +} +</th1></div> ADDED skins/default2/info.txt Index: skins/default2/info.txt ================================================================== --- skins/default2/info.txt +++ skins/default2/info.txt @@ -0,0 +1,4 @@ +Title: Khaki, No Logo +Description: A tan theme with the project title above the user identification + and no logo image. +Author: Unknown ADDED skins/default2/style.css Index: skins/default2/style.css ================================================================== --- skins/default2/style.css +++ skins/default2/style.css @@ -0,0 +1,137 @@ +/* General settings for the entire page */ +body { + margin: 0ex 0ex; + padding: 0px; + background-color: #fef3bc; + font-family: sans-serif; +} + +/* The project logo in the upper left-hand corner of each page */ +div.logo { + display: inline; + text-align: center; + vertical-align: bottom; + font-weight: bold; + font-size: 2.5em; + color: #a09048; +} + +/* The page title centered at the top of each page */ +div.title { + display: table-cell; + font-size: 2em; + font-weight: bold; + text-align: left; + padding: 0 0 0 5px; + color: #a09048; + vertical-align: bottom; + width: 100%; +} + +/* The login status message in the top right-hand corner */ +div.status { + display: table-cell; + text-align: right; + vertical-align: bottom; + color: #a09048; + padding: 5px 5px 0 0; + font-size: 0.8em; + font-weight: bold; +} + +/* The header across the top of the page */ +div.header { + display: table; + width: 100%; +} + +/* The main menu bar that appears at the top of the page beneath +** the header */ +div.mainmenu { + padding: 5px 10px 5px 10px; + font-size: 0.9em; + font-weight: bold; + text-align: center; + letter-spacing: 1px; + background-color: #a09048; + color: black; +} + +/* The submenu bar that *sometimes* appears below the main menu */ +div.submenu { + padding: 3px 10px 3px 0px; + font-size: 0.9em; + text-align: center; + background-color: #c0af58; + color: white; +} +div.mainmenu a, div.mainmenu a:visited, div.submenu a, div.submenu a:visited { + padding: 3px 10px 3px 10px; + color: white; + text-decoration: none; +} +div.mainmenu a:hover, div.submenu a:hover { + color: #a09048; + background-color: white; +} + +/* All page content from the bottom of the menu or submenu down to +** the footer */ +div.content { + padding: 1ex 5px; +} +div.content a { color: #706532; } +div.content a:link { color: #706532; } +div.content a:visited { color: #704032; } +div.content a:hover { background-color: white; color: #706532; } + +/* Some pages have section dividers */ +div.section { + margin-bottom: 0px; + margin-top: 1em; + padding: 3px 3px 0 3px; + font-size: 1.2em; + font-weight: bold; + background-color: #a09048; + color: white; +} + +/* The "Date" that occurs on the left hand side of timelines */ +div.divider { + background: #e1d498; + border: 2px #a09048 solid; + font-size: 1em; font-weight: normal; + padding: .25em; + margin: .2em 0 .2em 0; + float: left; + clear: left; +} + +/* The footer at the very bottom of the page */ +div.footer { + font-size: 0.8em; + margin-top: 12px; + padding: 5px 10px 5px 10px; + text-align: right; + background-color: #a09048; + color: white; +} + +/* Hyperlink colors */ +div.footer a { color: white; } +div.footer a:link { color: white; } +div.footer a:visited { color: white; } +div.footer a:hover { background-color: white; color: #558195; } + +/* <verbatim> blocks */ +pre.verbatim { + background-color: #f5f5f5; + padding: 0.5em; +} + +/* The label/value pairs on (for example) the ci page */ +table.label-value th { + vertical-align: top; + text-align: right; + padding: 0.2ex 2ex; +} ADDED skins/default3/footer.html Index: skins/default3/footer.html ================================================================== --- skins/default3/footer.html +++ skins/default3/footer.html @@ -0,0 +1,5 @@ +</div> +<div class="footer"> +Fossil version $manifest_version $manifest_date +</div> +</body></html> ADDED skins/default3/header.html Index: skins/default3/header.html ================================================================== --- skins/default3/header.html +++ skins/default3/header.html @@ -0,0 +1,54 @@ +<html> +<head> +<title>$<project_name>: $<title> + + + + +

+ +
$ +

$</div> + <div class="status"><nobr><th1> + if {[info exists login]} { + puts "Logged in as $login" + } else { + puts "Not logged in" + } + </th1></nobr></div> +</div> +<div class="mainmenu"><ul><th1> +html "<li><a href=''$baseurl$index_page''>Home</a></li>" +if {[anycap jor]} { + html "<li><a href=''$baseurl/timeline''>Timeline</a></li>" +} +if {[hascap oh]} { + html "<li><a href=''$baseurl/dir?ci=tip''>Files</a></li>" +} +if {[hascap o]} { + html "<li><a href=''$baseurl/leaves''>Leaves</a></li>" + html "<li><a href=''$baseurl/brlist''>Branches</a></li>" + html "<li><a href=''$baseurl/taglist''>Tags</a></li>" +} +if {[hascap r]} { + html "<li><a href=''$baseurl/reportlist''>Tickets</a></li>" +} +if {[hascap j]} { + html "<li><a href=''$baseurl/wiki''>Wiki</a></li>" +} +if {[hascap s]} { + html "<li><a href=''$baseurl/setup''>Admin</a></li>" +} elseif {[hascap a]} { + html "<li><a href=''$baseurl/setup_ulist''>Users</a></li>" +} +if {[info exists login]} { + html "<li><a href=''$baseurl/login''>Logout</a></li>" +} else { + html "<li><a href=''$baseurl/login''>Login</a></li>" +} +</th1></ul></div> +<div id="container"> ADDED skins/default3/info.txt Index: skins/default3/info.txt ================================================================== --- skins/default3/info.txt +++ skins/default3/info.txt @@ -0,0 +1,5 @@ +Title: Black & White, Menu on Left, No Logo +Description: Black letters on a white or cream background with the main menu + stuck on the left-hand side. +Author: Unknown + ADDED skins/default3/style.css Index: skins/default3/style.css ================================================================== --- skins/default3/style.css +++ skins/default3/style.css @@ -0,0 +1,170 @@ +/* General settings for the entire page */ +body { + margin:0px 0px 0px 0px; + padding:0px; + font-family:verdana, arial, helvetica, "sans serif"; + color:#333; + background-color:white; +} + +/* consistent colours */ +h2 { + color: #333; +} +h3 { + color: #333; +} + +/* The project logo in the upper left-hand corner of each page */ +div.logo { + display: table-cell; + text-align: left; + vertical-align: bottom; + font-weight: bold; + color: #333; +} + +/* The page title centered at the top of each page */ +div.title { + display: table-cell; + font-size: 2em; + font-weight: bold; + text-align: center; + color: #333; + vertical-align: bottom; + width: 100%; +} + +/* The login status message in the top right-hand corner */ +div.status { + display: table-cell; + padding-right: 10px; + text-align: right; + vertical-align: bottom; + padding-bottom: 5px; + color: #333; + font-size: 0.8em; + font-weight: bold; +} + +/* The header across the top of the page */ +div.header { + margin:10px 0px 10px 0px; + padding:1px 0px 0px 20px; + border-style:solid; + border-color:black; + border-width:1px 0px; + background-color:#eee; +} + +/* The main menu bar that appears at the top left of the page beneath +** the header. Width must be co-ordinated with the container below */ +div.mainmenu { + float: left; + margin-left: 10px; + margin-right: 10px; + font-size: 0.9em; + font-weight: bold; + padding:5px; + background-color:#eee; + border:1px solid #999; + width:8em; +} + +/* Main menu is now a list */ +div.mainmenu ul { + padding: 0; + list-style:none; +} +div.mainmenu a, div.mainmenu a:visited{ + padding: 1px 10px 1px 10px; + color: #333; + text-decoration: none; +} +div.mainmenu a:hover { + color: #eee; + background-color: #333; +} + +/* Container for the sub-menu and content so they don''t spread +** out underneath the main menu */ +#container { + padding-left: 9em; +} + +/* The submenu bar that *sometimes* appears below the main menu */ +div.submenu { + padding: 3px 10px 3px 10px; + font-size: 0.9em; + text-align: center; + border:1px solid #999; + border-width:1px 0px; + background-color: #eee; + color: #333; +} +div.submenu a, div.submenu a:visited { + padding: 3px 10px 3px 10px; + color: #333; + text-decoration: none; +} +div.submenu a:hover { + color: #eee; + background-color: #333; +} + +/* All page content from the bottom of the menu or submenu down to +** the footer */ +div.content { + float: right; + padding: 2ex 1ex 0ex 2ex; +} + +/* Some pages have section dividers */ +div.section { + margin-bottom: 0px; + margin-top: 1em; + padding: 1px 1px 1px 1px; + font-size: 1.2em; + font-weight: bold; + border-style:solid; + border-color:#999; + border-width:1px 0px; + background-color: #eee; + color: #333; +} + +/* The "Date" that occurs on the left hand side of timelines */ +div.divider { + background: #eee; + border: 2px #999 solid; + font-size: 1em; font-weight: normal; + padding: .25em; + margin: .2em 0 .2em 0; + float: left; + clear: left; + color: #333 +} + +/* The footer at the very bottom of the page */ +div.footer { + font-size: 0.8em; + margin-top: 12px; + padding: 5px 10px 5px 10px; + text-align: right; + background-color: #eee; + color: #555; +} + +/* <verbatim> blocks */ +pre.verbatim { + background-color: #f5f5f5; + padding: 0.5em; +} + +/* The label/value pairs on (for example) the ci page */ +table.label-value th { + vertical-align: top; + text-align: right; + padding: 0.2ex 2ex; +} + Index: src/add.c ================================================================== --- src/add.c +++ src/add.c @@ -36,28 +36,26 @@ Blob pathname; const char *zPath; file_tree_name(zName, &pathname, 1); zPath = blob_str(&pathname); - if( strcmp(zPath, "manifest")==0 - || strcmp(zPath, "_FOSSIL_")==0 + if( strcmp(zPath, "_FOSSIL_")==0 || strcmp(zPath, "_FOSSIL_-journal")==0 || strcmp(zPath, "_FOSSIL_-wal")==0 || strcmp(zPath, "_FOSSIL_-shm")==0 || strcmp(zPath, ".fos")==0 || strcmp(zPath, ".fos-journal")==0 || strcmp(zPath, ".fos-wal")==0 || strcmp(zPath, ".fos-shm")==0 - || strcmp(zPath, "manifest.uuid")==0 || blob_compare(&pathname, pOmit)==0 ){ fossil_warning("cannot add %s", zPath); }else{ if( !file_is_simple_pathname(zPath) ){ fossil_fatal("filename contains illegal characters: %s", zPath); } -#ifdef __MINGW32__ +#if defined(_WIN32) if( db_exists("SELECT 1 FROM vfile" " WHERE pathname=%Q COLLATE nocase", zPath) ){ db_multi_exec("UPDATE vfile SET deleted=0" " WHERE pathname=%Q COLLATE nocase", zPath); } @@ -152,11 +150,11 @@ db_begin_transaction(); if( !file_tree_name(g.zRepositoryName, &repo, 0) ){ blob_zero(&repo); } db_multi_exec("CREATE TEMP TABLE sfile(x TEXT PRIMARY KEY)"); -#ifdef __MINGW32__ +#if defined(_WIN32) db_multi_exec( "CREATE INDEX IF NOT EXISTS vfile_pathname " " ON vfile(pathname COLLATE nocase)" ); #endif Index: src/allrepo.c ================================================================== --- src/allrepo.c +++ src/allrepo.c @@ -34,11 +34,11 @@ static char *quoteFilename(const char *zFilename){ int i, c; int needQuote = 0; for(i=0; (c = zFilename[i])!=0; i++){ if( c=='"' ) return 0; - if( isspace(c) ) needQuote = 1; + if( fossil_isspace(c) ) needQuote = 1; if( c=='\\' && zFilename[i+1]==0 ) return 0; if( c=='$' ) return 0; } if( needQuote ){ return mprintf("\"%s\"", zFilename); @@ -53,17 +53,20 @@ ** ** Usage: %fossil all (list|ls|pull|push|rebuild|sync) ** ** The ~/.fossil file records the location of all repositories for a ** user. This command performs certain operations on all repositories -** that can be useful before or after a period of disconnection operation. +** that can be useful before or after a period of disconnected operation. ** -** On Win32 systems, this file is located in %LOCALAPPDATA%, %APDDATA% -** or %HOMEPATH% and is named _fossil. +** On Win32 systems, the file is named "_fossil" and is located in +** %LOCALAPPDATA%, %APPDATA% or %HOMEPATH%. ** ** Available operations are: ** +** ignore Arguments are repositories that should be ignored +** by subsequent list, pull, push, rebuild, and sync. +** ** list | ls Display the location of all repositories ** ** pull Run a "pull" operation on all repositories ** ** push Run a "push" on all repositories @@ -72,11 +75,12 @@ ** ** sync Run a "sync" on all repositories ** ** Respositories are automatically added to the set of known repositories ** when one of the following commands against the repository: clone, info, -** pull, push, or sync +** pull, push, or sync. Even previously ignored repositories are added back +** to the list of repositories by these commands. */ void all_cmd(void){ int n; Stmt q; const char *zCmd; @@ -99,13 +103,22 @@ zCmd = "pull -autourl -R"; }else if( strncmp(zCmd, "rebuild", n)==0 ){ zCmd = "rebuild"; }else if( strncmp(zCmd, "sync", n)==0 ){ zCmd = "sync -autourl -R"; + }else if( strncmp(zCmd, "ignore", n)==0 ){ + int j; + db_begin_transaction(); + for(j=3; j<g.argc; j++){ + db_multi_exec("DELETE FROM global_config WHERE name GLOB 'repo:%q'", + g.argv[j]); + } + db_end_transaction(0); + return; }else{ fossil_fatal("\"all\" subcommand should be one of: " - "list ls push pull rebuild sync"); + "ignore list ls push pull rebuild sync"); } zFossil = quoteFilename(g.argv[0]); nMissing = 0; db_prepare(&q, "SELECT DISTINCT substr(name, 6) COLLATE nocase" @@ -125,11 +138,11 @@ } zQFilename = quoteFilename(zFilename); zSyscmd = mprintf("%s %s %s", zFossil, zCmd, zQFilename); printf("%s\n", zSyscmd); fflush(stdout); - portable_system(zSyscmd); + fossil_system(zSyscmd); free(zSyscmd); free(zQFilename); } /* If any repositories whose names appear in the ~/.fossil file could not Index: src/attach.c ================================================================== --- src/attach.c +++ src/attach.c @@ -75,20 +75,20 @@ zFilename = &zFilename[i+1]; i = -1; } } if( strlen(zTarget)==UUID_SIZE && validate16(zTarget,UUID_SIZE) ){ - zUrlTail = mprintf("tkt=%s&file=%t", zTarget, zFilename); + zUrlTail = mprintf("tkt=%s&file=%t", zTarget, zFilename); }else{ - zUrlTail = mprintf("page=%t&file=%t", zTarget, zFilename); + zUrlTail = mprintf("page=%t&file=%t", zTarget, zFilename); } @ @ <p><a href="/attachview?%s(zUrlTail)">%h(zFilename)</a> - @ [<a href="/attachdownload/%t(zFilename)?%s(zUrlTail)">download</a>]<br> - if( zComment ) while( isspace(zComment[0]) ) zComment++; + @ [<a href="/attachdownload/%t(zFilename)?%s(zUrlTail)">download</a>]<br /> + if( zComment ) while( fossil_isspace(zComment[0]) ) zComment++; if( zComment && zComment[0] ){ - @ %w(zComment)<br> + @ %w(zComment)<br /> } if( zPage==0 && zTkt==0 ){ if( zSrc==0 || zSrc[0]==0 ){ zSrc = "Deleted from"; }else { @@ -251,13 +251,13 @@ } zName += n; if( zName[0]==0 ) zName = "unknown"; blob_appendf(&manifest, "A %F %F %s\n", zName, zTarget, zUUID); zComment = PD("comment", ""); - while( isspace(zComment[0]) ) zComment++; + while( fossil_isspace(zComment[0]) ) zComment++; n = strlen(zComment); - while( n>0 && isspace(zComment[n-1]) ){ n--; } + while( n>0 && fossil_isspace(zComment[n-1]) ){ n--; } if( n>0 ){ blob_appendf(&manifest, "C %F\n", zComment); } zDate = db_text(0, "SELECT datetime('now')"); zDate[10] = 'T'; @@ -270,25 +270,25 @@ db_end_transaction(0); cgi_redirect(zFrom); } style_header("Add Attachment"); @ <h2>Add Attachment To %s(zTargetType)</h2> - @ <form action="%s(g.zBaseURL)/attachadd" method="POST" - @ enctype="multipart/form-data"> + @ <form action="%s(g.zBaseURL)/attachadd" method="post" + @ enctype="multipart/form-data"><div> @ File to Attach: - @ <input type="file" name="f" size="60"><br> - @ Description:<br> - @ <textarea name="comment" cols=80 rows=5 wrap="virtual"></textarea><br> + @ <input type="file" name="f" size="60" /><br /> + @ Description:<br /> + @ <textarea name="comment" cols="80" rows="5" wrap="virtual"></textarea><br /> if( zTkt ){ - @ <input type="hidden" name="tkt" value="%h(zTkt)"> - }else{ - @ <input type="hidden" name="page" value="%h(zPage)"> - } - @ <input type="hidden" name="from" value="%h(zFrom)"> - @ <input type="submit" name="ok" value="Add Attachment"> - @ <input type="submit" name="cancel" value="Cancel"> - @ </form> + @ <input type="hidden" name="tkt" value="%h(zTkt)" /> + }else{ + @ <input type="hidden" name="page" value="%h(zPage)" /> + } + @ <input type="hidden" name="from" value="%h(zFrom)" /> + @ <input type="submit" name="ok" value="Add Attachment" /> + @ <input type="submit" name="cancel" value="Cancel" /> + @ </div></form> style_footer(); } /* @@ -349,21 +349,21 @@ manifest_crosslink(rid, &manifest); db_end_transaction(0); cgi_redirect(zFrom); } style_header("Delete Attachment"); - @ <form action="%s(g.zBaseURL)/attachdelete" method="POST"> + @ <form action="%s(g.zBaseURL)/attachdelete" method="post"><div> @ <p>Confirm that you want to delete the attachment named - @ "%h(zFile)" on %s(zTkt?"ticket":"wiki page") %h(zTarget):<br> + @ "%h(zFile)" on %s(zTkt?"ticket":"wiki page") %h(zTarget):<br /></p> if( zTkt ){ - @ <input type="hidden" name="tkt" value="%h(zTkt)"> + @ <input type="hidden" name="tkt" value="%h(zTkt)" /> }else{ - @ <input type="hidden" name="page" value="%h(zPage)"> + @ <input type="hidden" name="page" value="%h(zPage)" /> } - @ <input type="hidden" name="file" value="%h(zFile)"> - @ <input type="hidden" name="from" value="%h(zFrom)"> - @ <input type="submit" name="confirm" value="Delete"> - @ <input type="submit" name="cancel" value="Cancel"> - @ </form> + @ <input type="hidden" name="file" value="%h(zFile)" /> + @ <input type="hidden" name="from" value="%h(zFrom)" /> + @ <input type="submit" name="confirm" value="Delete" /> + @ <input type="submit" name="cancel" value="Cancel" /> + @ </div></form> style_footer(); } Index: src/bag.c ================================================================== --- src/bag.c +++ src/bag.c @@ -86,11 +86,11 @@ int nDel = 0; /* Number of deleted entries */ int nLive = 0; /* Number of live entries */ old = *p; assert( newSize>old.cnt ); - p->a = malloc( sizeof(p->a[0])*newSize ); + p->a = fossil_malloc( sizeof(p->a[0])*newSize ); p->sz = newSize; memset(p->a, 0, sizeof(p->a[0])*newSize ); for(i=0; i<old.sz; i++){ int e = old.a[i]; if( e>0 ){ Index: src/blob.c ================================================================== --- src/blob.c +++ src/blob.c @@ -72,25 +72,42 @@ /* ** We find that the built-in isspace() function does not work for ** some international character sets. So here is a substitute. */ -static int blob_isspace(char c){ +int fossil_isspace(char c){ return c==' ' || (c<='\r' && c>='\t'); } + +/* +** Other replacements for ctype.h functions. +*/ +int fossil_islower(char c){ return c>='a' && c<='z'; } +int fossil_isupper(char c){ return c>='A' && c<='Z'; } +int fossil_isdigit(char c){ return c>='0' && c<='9'; } +int fossil_tolower(char c){ + return fossil_isupper(c) ? c - 'A' + 'a' : c; +} +int fossil_isalpha(char c){ + return (c>='a' && c<='z') || (c>='A' && c<='Z'); +} +int fossil_isalnum(char c){ + return (c>='a' && c<='z') || (c>='A' && c<='Z') || (c>='0' && c<='9'); +} + /* ** COMMAND: test-isspace */ void isspace_cmd(void){ int i; for(i=0; i<=255; i++){ if( i==' ' || i=='\n' || i=='\t' || i=='\v' || i=='\f' || i=='\r' ){ - assert( blob_isspace((char)i) ); + assert( fossil_isspace((char)i) ); }else{ - assert( !blob_isspace((char)i) ); + assert( !fossil_isspace((char)i) ); } } printf("All 256 characters OK\n"); } @@ -119,12 +136,11 @@ pBlob->aData = 0; pBlob->nAlloc = 0; pBlob->nUsed = 0; pBlob->iCursor = 0; }else if( newSize>pBlob->nAlloc || newSize<pBlob->nAlloc-4000 ){ - char *pNew = realloc(pBlob->aData, newSize); - if( pNew==0 ) blob_panic(); + char *pNew = fossil_realloc(pBlob->aData, newSize); pBlob->aData = pNew; pBlob->nAlloc = newSize; if( pBlob->nUsed>pBlob->nAlloc ){ pBlob->nUsed = pBlob->nAlloc; } @@ -145,12 +161,11 @@ */ static void blobReallocStatic(Blob *pBlob, unsigned int newSize){ if( newSize==0 ){ *pBlob = empty_blob; }else{ - char *pNew = malloc( newSize ); - if( pNew==0 ) blob_panic(); + char *pNew = fossil_malloc( newSize ); if( pBlob->nUsed>newSize ) pBlob->nUsed = newSize; memcpy(pNew, pBlob->aData, pBlob->nUsed); pBlob->aData = pNew; pBlob->xRealloc = blobReallocMalloc; pBlob->nAlloc = newSize; @@ -429,11 +444,11 @@ ** not insert a new zero terminator. */ int blob_trim(Blob *p){ char *z = p->aData; int n = p->nUsed; - while( n>0 && blob_isspace(z[n-1]) ){ n--; } + while( n>0 && fossil_isspace(z[n-1]) ){ n--; } p->nUsed = n; return n; } /* @@ -452,15 +467,15 @@ */ int blob_token(Blob *pFrom, Blob *pTo){ char *aData = pFrom->aData; int n = pFrom->nUsed; int i = pFrom->iCursor; - while( i<n && blob_isspace(aData[i]) ){ i++; } + while( i<n && fossil_isspace(aData[i]) ){ i++; } pFrom->iCursor = i; - while( i<n && !blob_isspace(aData[i]) ){ i++; } + while( i<n && !fossil_isspace(aData[i]) ){ i++; } blob_extract(pFrom, i-pFrom->iCursor, pTo); - while( i<n && blob_isspace(aData[i]) ){ i++; } + while( i<n && fossil_isspace(aData[i]) ){ i++; } pFrom->iCursor = i; return pTo->nUsed; } /* @@ -523,11 +538,11 @@ int blob_is_int(Blob *pBlob, int *pValue){ const char *z = blob_buffer(pBlob); int i, n, c, v; n = blob_size(pBlob); v = 0; - for(i=0; i<n && (c = z[i])!=0 && isdigit(c); i++){ + for(i=0; i<n && (c = z[i])!=0 && c>='0' && c<='9'; i++){ v = v*10 + c - '0'; } if( i==n ){ *pValue = v; return 1; @@ -612,11 +627,11 @@ return blob_read_from_channel(pBlob, stdin, -1); } size = file_size(zFilename); blob_zero(pBlob); if( size<0 ){ - fossil_panic("no such file: %s", zFilename); + fossil_fatal("no such file: %s", zFilename); } if( size==0 ){ return 0; } blob_resize(pBlob, size); @@ -660,11 +675,11 @@ } nName = file_simplify_name(zName, nName); for(i=1; i<nName; i++){ if( zName[i]=='/' ){ zName[i] = 0; -#ifdef __MINGW32__ +#if defined(_WIN32) /* ** On Windows, local path looks like: C:/develop/project/file.txt ** The if stops us from trying to create a directory of a drive letter ** C: in this example. */ @@ -672,11 +687,11 @@ #endif if( file_mkdir(zName, 1) ){ fossil_fatal_recursive("unable to create directory %s", zName); return 0; } -#ifdef __MINGW32__ +#if defined(_WIN32) } #endif zName[i] = '/'; } } @@ -856,11 +871,11 @@ blob_reset(&b3); } printf("ok\n"); } -#ifdef __MINGW32__ +#if defined(_WIN32) /* ** Convert every \n character in the given blob into \r\n. */ void blob_add_cr(Blob *p){ char *z = p->aData; @@ -896,5 +911,26 @@ if( z[i]!='\r' ) z[j++] = z[i]; } z[j] = 0; p->nUsed = j; } + +/* +** Shell-escape the given string. Append the result to a blob. +*/ +void shell_escape(Blob *pBlob, const char *zIn){ + int n = blob_size(pBlob); + int k = strlen(zIn); + int i, c; + char *z; + for(i=0; (c = zIn[i])!=0; i++){ + if( fossil_isspace(c) || c=='"' || (c=='\\' && zIn[i+1]!=0) ){ + blob_appendf(pBlob, "\"%s\"", zIn); + z = blob_buffer(pBlob); + for(i=n+1; i<=n+k; i++){ + if( z[i]=='"' ) z[i] = '_'; + } + return; + } + } + blob_append(pBlob, zIn, -1); +} Index: src/branch.c ================================================================== --- src/branch.c +++ src/branch.c @@ -35,16 +35,19 @@ const char *zBranch; /* Name of the new branch */ char *zDate; /* Date that branch was created */ char *zComment; /* Check-in comment for the new branch */ const char *zColor; /* Color of the new branch */ Blob branch; /* manifest for the new branch */ - Blob parent; /* root check-in manifest */ - Manifest mParent; /* Parsed parent manifest */ + Manifest *pParent; /* Parsed parent manifest */ Blob mcksum; /* Self-checksum on the manifest */ + const char *zDateOvrd; /* Override date string */ + const char *zUserOvrd; /* Override user name */ noSign = find_option("nosign","",0)!=0; zColor = find_option("bgcolor","c",1); + zDateOvrd = find_option("date-override",0,1); + zUserOvrd = find_option("user-override",0,1); verify_all_options(); if( g.argc<5 ){ usage("new BRANCH-NAME CHECK-IN ?-bgcolor COLOR?"); } db_find_and_open_repository(1); @@ -67,41 +70,42 @@ db_begin_transaction(); rootid = name_to_rid(g.argv[4]); if( rootid==0 ){ fossil_fatal("unable to locate check-in off of which to branch"); } + + pParent = manifest_get(rootid, CFTYPE_MANIFEST); + if( pParent==0 ){ + fossil_fatal("%s is not a valid check-in", g.argv[4]); + } /* Create a manifest for the new branch */ blob_zero(&branch); + if( pParent->zBaseline ){ + blob_appendf(&branch, "B %s\n", pParent->zBaseline); + } zComment = mprintf("Create new branch named \"%h\"", zBranch); blob_appendf(&branch, "C %F\n", zComment); - zDate = db_text(0, "SELECT datetime('now')"); + zDate = date_in_standard_format(zDateOvrd ? zDateOvrd : "now"); zDate[10] = 'T'; blob_appendf(&branch, "D %s\n", zDate); /* Copy all of the content from the parent into the branch */ - content_get(rootid, &parent); - manifest_parse(&mParent, &parent); - if( mParent.type!=CFTYPE_MANIFEST ){ - fossil_fatal("%s is not a valid check-in", g.argv[4]); - } - for(i=0; i<mParent.nFile; ++i){ - if( mParent.aFile[i].zPerm[0] ){ - blob_appendf(&branch, "F %F %s %s\n", - mParent.aFile[i].zName, - mParent.aFile[i].zUuid, - mParent.aFile[i].zPerm); - }else{ - blob_appendf(&branch, "F %F %s\n", - mParent.aFile[i].zName, - mParent.aFile[i].zUuid); - } + for(i=0; i<pParent->nFile; ++i){ + blob_appendf(&branch, "F %F", pParent->aFile[i].zName); + if( pParent->aFile[i].zUuid ){ + blob_appendf(&branch, " %s", pParent->aFile[i].zUuid); + if( pParent->aFile[i].zPerm && pParent->aFile[i].zPerm[0] ){ + blob_appendf(&branch, " %s", pParent->aFile[i].zPerm); + } + } + blob_append(&branch, "\n", 1); } zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rootid); blob_appendf(&branch, "P %s\n", zUuid); - blob_appendf(&branch, "R %s\n", mParent.zRepoCksum); - manifest_clear(&mParent); + blob_appendf(&branch, "R %s\n", pParent->zRepoCksum); + manifest_destroy(pParent); /* Add the symbolic branch name and the "branch" tag to identify ** this as a new branch */ if( zColor!=0 ){ blob_appendf(&branch, "T *bgcolor * %F\n", zColor); @@ -120,11 +124,11 @@ const char *zTag = db_column_text(&q, 0); blob_appendf(&branch, "T -%F *\n", zTag); } db_finalize(&q); - blob_appendf(&branch, "U %F\n", g.zLogin); + blob_appendf(&branch, "U %F\n", zUserOvrd ? zUserOvrd : g.zLogin); md5sum_blob(&branch, &mcksum); blob_appendf(&branch, "Z %b\n", &mcksum); if( !noSign && clearsign(&branch, &branch) ){ Blob ans; blob_zero(&ans); @@ -215,73 +219,71 @@ ** Show a timeline of all branches */ void brlist_page(void){ Stmt q; int cnt; + int showClosed = P("closed")!=0; login_check_credentials(); if( !g.okRead ){ login_needed(); return; } - style_header("Branches"); + style_header(showClosed ? "Closed Branches" : "Open Branches"); style_submenu_element("Timeline", "Timeline", "brtimeline"); + if( showClosed ){ + style_submenu_element("Open","Open","brlist"); + }else{ + style_submenu_element("Closed","Closed","brlist?closed"); + } login_anonymous_available(); compute_leaves(0, 1); style_sidebox_begin("Nomenclature:", "33%"); @ <ol> - @ <li> An <b>open branch</b> is a branch that has one or + @ <li> An <div class="sideboxDescribed"><a href="brlist"> + @ open branch</a></div> is a branch that has one or @ more <a href="leaves">open leaves.</a> @ The presence of open leaves presumably means @ that the branch is still being extended with new check-ins.</li> - @ <li> A <b>closed branch</b> is a branch with only - @ <a href="leaves?closed">closed leaves</a>. + @ <li> A <div class="sideboxDescribed"><a href="brlist?closed"> + @ closed branch</a></div> is a branch with only + @ <div class="sideboxDescribed"><a href="leaves?closed"> + @ closed leaves</a></div>. @ Closed branches are fixed and do not change (unless they are first @ reopened)</li> @ </ol> style_sidebox_end(); - db_prepare(&q, - "SELECT DISTINCT value FROM tagxref" - " WHERE tagid=%d AND value NOT NULL" - " AND rid IN leaves" - " ORDER BY value /*sort*/", - TAG_BRANCH - ); - cnt = 0; - while( db_step(&q)==SQLITE_ROW ){ - const char *zBr = db_column_text(&q, 0); - if( cnt==0 ){ - @ <h2>Open Branches:</h2> - @ <ul> - cnt++; - } - if( g.okHistory ){ - @ <li><a href="%s(g.zBaseURL)/timeline?r=%T(zBr)">%h(zBr)</a></li> - }else{ - @ <li><b>%h(zBr)</b></li> - } - } - db_finalize(&q); - if( cnt ){ - @ </ul> - } - cnt = 0; - db_prepare(&q, - "SELECT value FROM tagxref" - " WHERE tagid=%d AND value NOT NULL" - " EXCEPT " - "SELECT value FROM tagxref" - " WHERE tagid=%d AND value NOT NULL" - " AND rid IN leaves" - " ORDER BY value /*sort*/", - TAG_BRANCH, TAG_BRANCH - ); - while( db_step(&q)==SQLITE_ROW ){ - const char *zBr = db_column_text(&q, 0); - if( cnt==0 ){ - @ <h2>Closed Branches:</h2> - @ <ul> - cnt++; + cnt = 0; + if( !showClosed ){ + db_prepare(&q, + "SELECT DISTINCT value FROM tagxref" + " WHERE tagid=%d AND value NOT NULL" + " AND rid IN leaves" + " ORDER BY value /*sort*/", + TAG_BRANCH + ); + }else{ + db_prepare(&q, + "SELECT value FROM tagxref" + " WHERE tagid=%d AND value NOT NULL" + " EXCEPT " + "SELECT value FROM tagxref" + " WHERE tagid=%d AND value NOT NULL" + " AND rid IN leaves" + " ORDER BY value /*sort*/", + TAG_BRANCH, TAG_BRANCH + ); + } + while( db_step(&q)==SQLITE_ROW ){ + const char *zBr = db_column_text(&q, 0); + if( cnt==0 ){ + if( showClosed ){ + @ <h2>Closed Branches:</h2> + }else{ + @ <h2>Open Branches:</h2> + } + @ <ul> + cnt++; } if( g.okHistory ){ @ <li><a href="%s(g.zBaseURL)/timeline?r=%T(zBr)">%h(zBr)</a></li> }else{ @ <li><b>%h(zBr)</b></li> @@ -289,13 +291,11 @@ } if( cnt ){ @ </ul> } db_finalize(&q); - @ </ul> - @ <br clear="both"> - @ <script> + @ <script type="text/JavaScript"> @ function xin(id){ @ } @ function xout(id){ @ } @ </script> @@ -346,14 +346,13 @@ " ORDER BY event.mtime DESC", timeline_query_for_www(), TAG_BRANCH ); www_print_timeline(&q, 0, brtimeline_extra); db_finalize(&q); - @ <br clear="both"> - @ <script> + @ <script type="text/JavaScript"> @ function xin(id){ @ } @ function xout(id){ @ } @ </script> style_footer(); } Index: src/browse.c ================================================================== --- src/browse.c +++ src/browse.c @@ -71,19 +71,24 @@ ** There is no hyperlink on the file element of the path. ** ** The computed string is appended to the pOut blob. pOut should ** have already been initialized. */ -void hyperlinked_path(const char *zPath, Blob *pOut){ +void hyperlinked_path(const char *zPath, Blob *pOut, const char *zCI){ int i, j; char *zSep = ""; for(i=0; zPath[i]; i=j){ for(j=i; zPath[j] && zPath[j]!='/'; j++){} if( zPath[j] && g.okHistory ){ - blob_appendf(pOut, "%s<a href=\"%s/dir?name=%#T\">%#h</a>", - zSep, g.zBaseURL, j, zPath, j-i, &zPath[i]); + if( zCI ){ + blob_appendf(pOut, "%s<a href=\"%s/dir?ci=%S&name=%#T\">%#h</a>", + zSep, g.zBaseURL, zCI, j, zPath, j-i, &zPath[i]); + }else{ + blob_appendf(pOut, "%s<a href=\"%s/dir?name=%#T\">%#h</a>", + zSep, g.zBaseURL, j, zPath, j-i, &zPath[i]); + } }else{ blob_appendf(pOut, "%s%#h", zSep, j-i, &zPath[i]); } zSep = "/"; while( zPath[j]=='/' ){ j++; } @@ -99,20 +104,21 @@ ** name=PATH Directory to display. Required. ** ci=LABEL Show only files in this check-in. Optional. */ void page_dir(void){ const char *zD = P("name"); + int nD = zD ? strlen(zD)+1 : 0; int mxLen; int nCol, nRow; int cnt, i; char *zPrefix; Stmt q; const char *zCI = P("ci"); int rid = 0; - Blob content; + char *zUuid = 0; Blob dirname; - Manifest m; + Manifest *pM = 0; const char *zSubdirLink; login_check_credentials(); if( !g.okHistory ){ login_needed(); return; } style_header("File List"); @@ -120,54 +126,68 @@ pathelementFunc, 0, 0); /* If the name= parameter is an empty string, make it a NULL pointer */ if( zD && strlen(zD)==0 ){ zD = 0; } - /* If a specific check-in is requested, fetch and parse it. */ - if( zCI && (rid = name_to_rid(zCI))!=0 && content_get(rid, &content) ){ - if( !manifest_parse(&m, &content) || m.type!=CFTYPE_MANIFEST ){ + /* If a specific check-in is requested, fetch and parse it. If the + ** specific check-in does not exist, clear zCI. zCI==0 will cause all + ** files from all check-ins to be displayed. + */ + if( zCI ){ + pM = manifest_get_by_name(zCI, &rid); + if( pM ){ + zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rid); + }else{ zCI = 0; } } + /* Compute the title of the page */ blob_zero(&dirname); if( zD ){ blob_append(&dirname, "in directory ", -1); - hyperlinked_path(zD, &dirname); + hyperlinked_path(zD, &dirname, zCI); zPrefix = mprintf("%h/", zD); }else{ blob_append(&dirname, "in the top-level directory", -1); zPrefix = ""; } if( zCI ){ - char *zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rid); char zShort[20]; memcpy(zShort, zUuid, 10); zShort[10] = 0; @ <h2>Files of check-in [<a href="vinfo?name=%T(zUuid)">%s(zShort)</a>] @ %s(blob_str(&dirname))</h2> - zSubdirLink = mprintf("%s/dir?ci=%S&name=%T", g.zTop, zUuid, zPrefix); + zSubdirLink = mprintf("%s/dir?ci=%S&name=%T", g.zTop, zUuid, zPrefix); if( zD ){ style_submenu_element("Top", "Top", "%s/dir?ci=%S", g.zTop, zUuid); style_submenu_element("All", "All", "%s/dir?name=%t", g.zTop, zD); }else{ style_submenu_element("All", "All", "%s/dir", g.zBaseURL); } }else{ + int hasTrunk; @ <h2>The union of all files from all check-ins @ %s(blob_str(&dirname))</h2> + hasTrunk = db_exists( + "SELECT 1 FROM tagxref WHERE tagid=%d AND value='trunk'", + TAG_BRANCH); zSubdirLink = mprintf("%s/dir?name=%T", g.zBaseURL, zPrefix); if( zD ){ style_submenu_element("Top", "Top", "%s/dir", g.zBaseURL); - style_submenu_element("Tip", "Tip", "%s/dir?name=%t&ci=tip", + style_submenu_element("Tip", "Tip", "%s/dir?name=%t&ci=tip", g.zBaseURL, zD); - style_submenu_element("Trunk", "Trunk", "%s/dir?name=%t&ci=trunk", - g.zBaseURL,zD); + if( hasTrunk ){ + style_submenu_element("Trunk", "Trunk", "%s/dir?name=%t&ci=trunk", + g.zBaseURL,zD); + } }else{ style_submenu_element("Tip", "Tip", "%s/dir?ci=tip", g.zBaseURL); - style_submenu_element("Trunk", "Trunk", "%s/dir?ci=trunk", g.zBaseURL); + if( hasTrunk ){ + style_submenu_element("Trunk", "Trunk", "%s/dir?ci=trunk", g.zBaseURL); + } } } /* Compute the temporary table "localfiles" containing the names ** of all files and subdirectories in the zD[] directory. @@ -176,39 +196,47 @@ ** first and it also gives us an easy way to distinguish files ** from directories in the loop that follows. */ db_multi_exec( "CREATE TEMP TABLE localfiles(x UNIQUE NOT NULL, u);" - "CREATE TEMP TABLE allfiles(x UNIQUE NOT NULL, u);" ); if( zCI ){ Stmt ins; - int i; - db_prepare(&ins, "INSERT INTO allfiles VALUES(:x, :u)"); - for(i=0; i<m.nFile; i++){ - db_bind_text(&ins, ":x", m.aFile[i].zName); - db_bind_text(&ins, ":u", m.aFile[i].zUuid); - db_step(&ins); - db_reset(&ins); - } - db_finalize(&ins); - }else{ - db_multi_exec( - "INSERT INTO allfiles SELECT name, NULL FROM filename" - ); - } - if( zD ){ - db_multi_exec( - "INSERT OR IGNORE INTO localfiles " - " SELECT pathelement(x,%d), u FROM allfiles" - " WHERE x GLOB '%q/*'", - strlen(zD)+1, zD - ); - }else{ - db_multi_exec( - "INSERT OR IGNORE INTO localfiles " - " SELECT pathelement(x,0), u FROM allfiles" + ManifestFile *pFile; + ManifestFile *pPrev = 0; + int nPrev = 0; + int c; + + db_prepare(&ins, + "INSERT OR IGNORE INTO localfiles VALUES(pathelement(:x,0), :u)" + ); + manifest_file_rewind(pM); + while( (pFile = manifest_file_next(pM,0))!=0 ){ + if( nD>0 && memcmp(pFile->zName, zD, nD-1)!=0 ) continue; + if( pPrev && memcmp(&pFile->zName[nD],&pPrev->zName[nD],nPrev)==0 ){ + continue; + } + db_bind_text(&ins, ":x", &pFile->zName[nD]); + db_bind_text(&ins, ":u", pFile->zUuid); + db_step(&ins); + db_reset(&ins); + pPrev = pFile; + for(nPrev=0; (c=pPrev->zName[nD+nPrev]) && c!='/'; nPrev++){} + if( c=='/' ) nPrev++; + } + db_finalize(&ins); + }else if( zD ){ + db_multi_exec( + "INSERT OR IGNORE INTO localfiles" + " SELECT pathelement(name,%d), NULL FROM filename" + " WHERE name GLOB '%q/*'", + nD, zD + ); + }else{ + db_multi_exec( + "INSERT OR IGNORE INTO localfiles" + " SELECT pathelement(name,0), NULL FROM filename" ); } /* Generate a multi-column table listing the contents of zD[] ** directory. @@ -216,29 +244,31 @@ mxLen = db_int(12, "SELECT max(length(x)) FROM localfiles /*scan*/"); cnt = db_int(0, "SELECT count(*) FROM localfiles /*scan*/"); nCol = 4; nRow = (cnt+nCol-1)/nCol; db_prepare(&q, "SELECT x, u FROM localfiles ORDER BY x /*scan*/"); - @ <table border="0" width="100%%"><tr><td valign="top" width="25%%"> + @ <table class="browser"><tr><td class="browser"><ul class="browser"> i = 0; while( db_step(&q)==SQLITE_ROW ){ const char *zFN; if( i==nRow ){ - @ </td><td valign="top" width="25%%"> + @ </ul></td><td class="browser"><ul class="browser"> i = 0; } i++; zFN = db_column_text(&q, 0); if( zFN[0]=='/' ){ zFN++; @ <li><a href="%s(zSubdirLink)%T(zFN)">%h(zFN)/</a></li> }else if( zCI ){ const char *zUuid = db_column_text(&q, 1); - @ <li><a href="%s(g.zBaseURL)/artifact?name=%s(zUuid)">%h(zFN)</a> + @ <li><a href="%s(g.zBaseURL)/artifact?name=%s(zUuid)">%h(zFN)</a></li> }else{ - @ <li><a href="%s(g.zBaseURL)/finfo?name=%T(zPrefix)%T(zFN)">%h(zFN)</a> + @ <li><a href="%s(g.zBaseURL)/finfo?name=%T(zPrefix)%T(zFN)">%h(zFN) + @ </a></li> } } db_finalize(&q); - @ </td></tr></table> + manifest_destroy(pM); + @ </ul></td></tr></table> style_footer(); } Index: src/captcha.c ================================================================== --- src/captcha.c +++ src/captcha.c @@ -69,11 +69,11 @@ ** Render an 8-character hexadecimal string as ascii art. ** Space to hold the result is obtained from malloc() and should be freed ** by the caller. */ char *captcha_render(const char *zPw){ - char *z = malloc( 500 ); + char *z = fossil_malloc( 500 ); int i, j, k, m; k = 0; for(i=0; i<6; i++){ for(j=0; j<8; j++){ @@ -202,11 +202,11 @@ ** Render an 8-digit hexadecimal string as ascii arg. ** Space to hold the result is obtained from malloc() and should be freed ** by the caller. */ char *captcha_render(const char *zPw){ - char *z = malloc( 300 ); + char *z = fossil_malloc( 300 ); int i, j, k, m; const char *zChar; k = 0; for(i=0; i<4; i++){ @@ -359,11 +359,11 @@ ** Render an 8-digit hexadecimal string as ascii arg. ** Space to hold the result is obtained from malloc() and should be freed ** by the caller. */ char *captcha_render(const char *zPw){ - char *z = malloc( 600 ); + char *z = fossil_malloc( 600 ); int i, j, k, m; const char *zChar; k = 0; for(i=0; i<6; i++){ @@ -399,11 +399,11 @@ } } /* ** Compute a seed value for a captcha. The seed is public and is sent -** has a hidden parameter with the page that contains the captcha. Knowledge +** as a hidden parameter with the page that contains the captcha. Knowledge ** of the seed is insufficient for determining the captcha without additional ** information held only on the server and never revealed. */ unsigned int captcha_seed(void){ unsigned int x; Index: src/cgi.c ================================================================== --- src/cgi.c +++ src/cgi.c @@ -20,15 +20,16 @@ ** dispensing QUERY_STRING parameters and cookies, the "mprintf()" ** formatting function and its cousins, and routines to encode and ** decode strings in HTML or HTTP. */ #include "config.h" -#ifdef __MINGW32__ +#ifdef _WIN32 # include <windows.h> /* for Sleep once server works again */ -# include <winsock2.h> /* socket operations */ -# define sleep Sleep /* windows does not have sleep, but Sleep */ -# include <ws2tcpip.h> +# if defined(__MINGW32__) +# define sleep Sleep /* windows does not have sleep, but Sleep */ +# include <ws2tcpip.h> +# endif #else # include <sys/socket.h> # include <netinet/in.h> # include <arpa/inet.h> # include <sys/times.h> @@ -41,10 +42,13 @@ #endif #include <time.h> #include <stdio.h> #include <stdlib.h> #include <unistd.h> +#if defined (__POCC__) +# undef INTERFACE +#endif #include "cgi.h" #if INTERFACE /* ** Shortcuts for cgi_parameter. P("x") returns the value of query parameter @@ -383,12 +387,11 @@ ** deallocated after this routine returns. */ void cgi_set_parameter_nocopy(const char *zName, const char *zValue){ if( nAllocQP<=nUsedQP ){ nAllocQP = nAllocQP*2 + 10; - aParamQP = realloc( aParamQP, nAllocQP*sizeof(aParamQP[0]) ); - if( aParamQP==0 ) fossil_exit(1); + aParamQP = fossil_realloc( aParamQP, nAllocQP*sizeof(aParamQP[0]) ); } aParamQP[nUsedQP].zName = zName; aParamQP[nUsedQP].zValue = zValue; if( g.fHttpTrace ){ fprintf(stderr, "# cgi: %s = [%s]\n", zName, zValue); @@ -459,11 +462,11 @@ */ static void add_param_list(char *z, int terminator){ while( *z ){ char *zName; char *zValue; - while( isspace(*z) ){ z++; } + while( fossil_isspace(*z) ){ z++; } zName = z; while( *z && *z!='=' && *z!=terminator ){ z++; } if( *z=='=' ){ *z = 0; z++; @@ -476,11 +479,11 @@ dehttpize(zValue); }else{ if( *z ){ *z++ = 0; } zValue = ""; } - if( islower(zName[0]) ){ + if( fossil_islower(zName[0]) ){ cgi_set_parameter_nocopy(zName, zValue); } } } @@ -571,11 +574,11 @@ ** in the example above. */ static int tokenize_line(char *z, int mxArg, char **azArg){ int i = 0; while( *z ){ - while( isspace(*z) || *z==';' ){ z++; } + while( fossil_isspace(*z) || *z==';' ){ z++; } if( *z=='"' && z[1] ){ *z = 0; z++; if( i<mxArg-1 ){ azArg[i++] = z; } while( *z && *z!='"' ){ z++; } @@ -582,11 +585,11 @@ if( *z==0 ) break; *z = 0; z++; }else{ if( i<mxArg-1 ){ azArg[i++] = z; } - while( *z && !isspace(*z) && *z!=';' && *z!='"' ){ z++; } + while( *z && !fossil_isspace(*z) && *z!=';' && *z!='"' ){ z++; } if( *z && *z!='"' ){ *z = 0; z++; } } @@ -617,11 +620,11 @@ if( zBoundry==0 ) return; while( (zLine = get_line_from_string(&z, &len))!=0 ){ if( zLine[0]==0 ){ int nContent = 0; zValue = get_bounded_content(&z, &len, zBoundry, &nContent); - if( zName && zValue && islower(zName[0]) ){ + if( zName && zValue && fossil_islower(zName[0]) ){ cgi_set_parameter_nocopy(zName, zValue); if( showBytes ){ cgi_set_parameter_nocopy(mprintf("%s:bytes", zName), mprintf("%d",nContent)); } @@ -629,25 +632,25 @@ zName = 0; showBytes = 0; }else{ nArg = tokenize_line(zLine, sizeof(azArg)/sizeof(azArg[0]), azArg); for(i=0; i<nArg; i++){ - int c = tolower(azArg[i][0]); + int c = fossil_tolower(azArg[i][0]); int n = strlen(azArg[i]); if( c=='c' && sqlite3_strnicmp(azArg[i],"content-disposition:",n)==0 ){ i++; }else if( c=='n' && sqlite3_strnicmp(azArg[i],"name=",n)==0 ){ zName = azArg[++i]; }else if( c=='f' && sqlite3_strnicmp(azArg[i],"filename=",n)==0 ){ char *z = azArg[++i]; - if( zName && z && islower(zName[0]) ){ + if( zName && z && fossil_islower(zName[0]) ){ cgi_set_parameter_nocopy(mprintf("%s:filename",zName), z); } showBytes = 1; }else if( c=='c' && sqlite3_strnicmp(azArg[i],"content-type:",n)==0 ){ char *z = azArg[++i]; - if( zName && z && islower(zName[0]) ){ + if( zName && z && fossil_islower(zName[0]) ){ cgi_set_parameter_nocopy(mprintf("%s:mimetype",zName), z); } } } } @@ -677,12 +680,11 @@ g.zContentType = zType = P("CONTENT_TYPE"); if( len>0 && zType ){ blob_zero(&g.cgiIn); if( strcmp(zType,"application/x-www-form-urlencoded")==0 || strncmp(zType,"multipart/form-data",19)==0 ){ - z = malloc( len+1 ); - if( z==0 ) fossil_exit(1); + z = fossil_malloc( len+1 ); len = fread(z, 1, len, g.httpIn); z[len] = 0; if( zType[0]=='a' ){ add_param_list(z, '&'); }else{ @@ -769,11 +771,11 @@ /* If no match is found and the name begins with an upper-case ** letter, then check to see if there is an environment variable ** with the given name. */ - if( isupper(zName[0]) ){ + if( fossil_isupper(zName[0]) ){ const char *zValue = getenv(zName); if( zValue ){ cgi_set_parameter_nocopy(zName, zValue); CGIDEBUG(("env-match [%s] = [%s]\n", zName, zValue)); return zValue; @@ -851,145 +853,10 @@ cgi_printf("%s = %s <br />\n", htmlize(aParamQP[i].zName, -1), htmlize(aParamQP[i].zValue, -1)); } } -/* -** Write HTML text for an option menu to standard output. zParam -** is the query parameter that the option menu sets. zDflt is the -** initial value of the option menu. Addition arguments are name/value -** pairs that define values on the menu. The list is terminated with -** a single NULL argument. -*/ -void cgi_optionmenu(int in, const char *zP, const char *zD, ...){ - va_list ap; - char *zName, *zVal; - int dfltSeen = 0; - cgi_printf("%*s<select size=1 name=\"%s\">\n", in, "", zP); - va_start(ap, zD); - while( (zName = va_arg(ap, char*))!=0 && (zVal = va_arg(ap, char*))!=0 ){ - if( strcmp(zVal,zD)==0 ){ dfltSeen = 1; break; } - } - va_end(ap); - if( !dfltSeen ){ - if( zD[0] ){ - cgi_printf("%*s<option value=\"%h\" selected>%h</option>\n", - in+2, "", zD, zD); - }else{ - cgi_printf("%*s<option value=\"\" selected> </option>\n", in+2, ""); - } - } - va_start(ap, zD); - while( (zName = va_arg(ap, char*))!=0 && (zVal = va_arg(ap, char*))!=0 ){ - if( zName[0] ){ - cgi_printf("%*s<option value=\"%h\"%s>%h</option>\n", - in+2, "", - zVal, - strcmp(zVal, zD) ? "" : " selected", - zName - ); - }else{ - cgi_printf("%*s<option value=\"\"%s> </option>\n", - in+2, "", - strcmp(zVal, zD) ? "" : " selected" - ); - } - } - va_end(ap); - cgi_printf("%*s</select>\n", in, ""); -} - -/* -** This routine works a lot like cgi_optionmenu() except that the list of -** values is contained in an array. Also, the values are just values, not -** name/value pairs as in cgi_optionmenu. -*/ -void cgi_v_optionmenu( - int in, /* Indent by this amount */ - const char *zP, /* The query parameter name */ - const char *zD, /* Default value */ - const char **az /* NULL-terminated list of allowed values */ -){ - const char *zVal; - int i; - cgi_printf("%*s<select size=1 name=\"%s\">\n", in, "", zP); - for(i=0; az[i]; i++){ - if( strcmp(az[i],zD)==0 ) break; - } - if( az[i]==0 ){ - if( zD[0]==0 ){ - cgi_printf("%*s<option value=\"\" selected> </option>\n", - in+2, ""); - }else{ - cgi_printf("%*s<option value=\"%h\" selected>%h</option>\n", - in+2, "", zD, zD); - } - } - while( (zVal = *(az++))!=0 ){ - if( zVal[0] ){ - cgi_printf("%*s<option value=\"%h\"%s>%h</option>\n", - in+2, "", - zVal, - strcmp(zVal, zD) ? "" : " selected", - zVal - ); - }else{ - cgi_printf("%*s<option value=\"\"%s> </option>\n", - in+2, "", - strcmp(zVal, zD) ? "" : " selected" - ); - } - } - cgi_printf("%*s</select>\n", in, ""); -} - -/* -** This routine works a lot like cgi_v_optionmenu() except that the list -** is a list of pairs. The first element of each pair is the value used -** internally and the second element is the value displayed to the user. -*/ -void cgi_v_optionmenu2( - int in, /* Indent by this amount */ - const char *zP, /* The query parameter name */ - const char *zD, /* Default value */ - const char **az /* NULL-terminated list of allowed values */ -){ - const char *zVal; - int i; - cgi_printf("%*s<select size=1 name=\"%s\">\n", in, "", zP); - for(i=0; az[i]; i+=2){ - if( strcmp(az[i],zD)==0 ) break; - } - if( az[i]==0 ){ - if( zD[0]==0 ){ - cgi_printf("%*s<option value=\"\" selected> </option>\n", - in+2, ""); - }else{ - cgi_printf("%*s<option value=\"%h\" selected>%h</option>\n", - in+2, "", zD, zD); - } - } - while( (zVal = *(az++))!=0 ){ - const char *zName = *(az++); - if( zName[0] ){ - cgi_printf("%*s<option value=\"%h\"%s>%h</option>\n", - in+2, "", - zVal, - strcmp(zVal, zD) ? "" : " selected", - zName - ); - }else{ - cgi_printf("%*s<option value=\"%h\"%s> </option>\n", - in+2, "", - zVal, - strcmp(zVal, zD) ? "" : " selected" - ); - } - } - cgi_printf("%*s</select>\n", in, ""); -} - /* ** This routine works like "printf" except that it has the ** extra formatting capabilities such as %h and %t. */ void cgi_printf(const char *zFormat, ...){ @@ -1047,17 +914,17 @@ char *zResult = 0; if( zInput==0 ){ if( zLeftOver ) *zLeftOver = 0; return 0; } - while( isspace(*zInput) ){ zInput++; } + while( fossil_isspace(*zInput) ){ zInput++; } zResult = zInput; - while( *zInput && !isspace(*zInput) ){ zInput++; } + while( *zInput && !fossil_isspace(*zInput) ){ zInput++; } if( *zInput ){ *zInput = 0; zInput++; - while( isspace(*zInput) ){ zInput++; } + while( fossil_isspace(*zInput) ){ zInput++; } } if( zLeftOver ){ *zLeftOver = zInput; } return zResult; } @@ -1072,11 +939,11 @@ */ void cgi_handle_http_request(const char *zIpAddr){ char *z, *zToken; int i; struct sockaddr_in remoteName; - size_t size = sizeof(struct sockaddr_in); + socklen_t size = sizeof(struct sockaddr_in); char zLine[2000]; /* A single line of input. */ g.fullHttpReply = 1; if( fgets(zLine, sizeof(zLine),g.httpIn)==0 ){ malformed_request(); @@ -1100,11 +967,11 @@ if( zToken[i] ) zToken[i++] = 0; cgi_setenv("PATH_INFO", zToken); cgi_setenv("QUERY_STRING", &zToken[i]); if( zIpAddr==0 && getpeername(fileno(g.httpIn), (struct sockaddr*)&remoteName, - (socklen_t*)&size)>=0 + &size)>=0 ){ zIpAddr = inet_ntoa(remoteName.sin_addr); } if( zIpAddr ){ cgi_setenv("REMOTE_ADDR", zIpAddr); @@ -1117,37 +984,52 @@ char *zFieldName; char *zVal; zFieldName = extract_token(zLine,&zVal); if( zFieldName==0 || *zFieldName==0 ) break; - while( isspace(*zVal) ){ zVal++; } + while( fossil_isspace(*zVal) ){ zVal++; } i = strlen(zVal); - while( i>0 && isspace(zVal[i-1]) ){ i--; } + while( i>0 && fossil_isspace(zVal[i-1]) ){ i--; } zVal[i] = 0; - for(i=0; zFieldName[i]; i++){ zFieldName[i] = tolower(zFieldName[i]); } - if( strcmp(zFieldName,"user-agent:")==0 ){ - cgi_setenv("HTTP_USER_AGENT", zVal); - }else if( strcmp(zFieldName,"content-length:")==0 ){ + for(i=0; zFieldName[i]; i++){ + zFieldName[i] = fossil_tolower(zFieldName[i]); + } + if( strcmp(zFieldName,"content-length:")==0 ){ cgi_setenv("CONTENT_LENGTH", zVal); - }else if( strcmp(zFieldName,"referer:")==0 ){ - cgi_setenv("HTTP_REFERER", zVal); - }else if( strcmp(zFieldName,"host:")==0 ){ - cgi_setenv("HTTP_HOST", zVal); }else if( strcmp(zFieldName,"content-type:")==0 ){ cgi_setenv("CONTENT_TYPE", zVal); }else if( strcmp(zFieldName,"cookie:")==0 ){ cgi_setenv("HTTP_COOKIE", zVal); + }else if( strcmp(zFieldName,"https:")==0 ){ + cgi_setenv("HTTPS", zVal); + }else if( strcmp(zFieldName,"host:")==0 ){ + cgi_setenv("HTTP_HOST", zVal); }else if( strcmp(zFieldName,"if-none-match:")==0 ){ cgi_setenv("HTTP_IF_NONE_MATCH", zVal); }else if( strcmp(zFieldName,"if-modified-since:")==0 ){ cgi_setenv("HTTP_IF_MODIFIED_SINCE", zVal); } +#if 0 + else if( strcmp(zFieldName,"referer:")==0 ){ + cgi_setenv("HTTP_REFERER", zVal); + }else if( strcmp(zFieldName,"user-agent:")==0 ){ + cgi_setenv("HTTP_USER_AGENT", zVal); + } +#endif } cgi_init(); } +#if INTERFACE +/* +** Bitmap values for the flags parameter to cgi_http_server(). +*/ +#define HTTP_SERVER_LOCALHOST 0x0001 /* Bind to 127.0.0.1 only */ + +#endif /* INTERFACE */ + /* ** Maximum number of child processes that we can have running ** at one time before we start slowing things down. */ #define MAX_PARALLEL 2 @@ -1160,19 +1042,19 @@ ** The parent never returns from this procedure. ** ** Return 0 to each child as it runs. If unable to establish a ** listening socket, return non-zero. */ -int cgi_http_server(int mnPort, int mxPort, char *zBrowser){ -#ifdef __MINGW32__ +int cgi_http_server(int mnPort, int mxPort, char *zBrowser, int flags){ +#if defined(_WIN32) /* Use win32_http_server() instead */ fossil_exit(1); #else int listener = -1; /* The server socket */ int connection; /* A socket for each individual connection */ fd_set readfds; /* Set of file descriptors for select() */ - size_t lenaddr; /* Length of the inaddr structure */ + socklen_t lenaddr; /* Length of the inaddr structure */ int child; /* PID of the child process */ int nchildren = 0; /* Number of child processes */ struct timeval delay; /* How long to wait inside select() */ struct sockaddr_in inaddr; /* The socket address */ int opt = 1; /* setsockopt flag */ @@ -1179,11 +1061,15 @@ int iPort = mnPort; while( iPort<=mxPort ){ memset(&inaddr, 0, sizeof(inaddr)); inaddr.sin_family = AF_INET; - inaddr.sin_addr.s_addr = INADDR_ANY; + if( flags & HTTP_SERVER_LOCALHOST ){ + inaddr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + }else{ + inaddr.sin_addr.s_addr = htonl(INADDR_ANY); + } inaddr.sin_port = htons(iPort); listener = socket(AF_INET, SOCK_STREAM, 0); if( listener<0 ){ iPort++; continue; @@ -1224,14 +1110,14 @@ } delay.tv_sec = 60; delay.tv_usec = 0; FD_ZERO(&readfds); FD_SET( listener, &readfds); - if( select( listener+1, &readfds, 0, 0, &delay) ){ + select( listener+1, &readfds, 0, 0, &delay); + if( FD_ISSET(listener, &readfds) ){ lenaddr = sizeof(inaddr); - connection = accept(listener, (struct sockaddr*)&inaddr, - (socklen_t*) &lenaddr); + connection = accept(listener, (struct sockaddr*)&inaddr, &lenaddr); if( connection>=0 ){ child = fork(); if( child!=0 ){ if( child>0 ) nchildren++; close(connection); Index: src/checkin.c ================================================================== --- src/checkin.c +++ src/checkin.c @@ -215,21 +215,21 @@ int cTerm; if( zGlobList==0 || zGlobList[0]==0 ) return "0"; blob_zero(&expr); while( zGlobList[0] ){ - while( isspace(zGlobList[0]) || zGlobList[0]==',' ) zGlobList++; + while( fossil_isspace(zGlobList[0]) || zGlobList[0]==',' ) zGlobList++; if( zGlobList[0]==0 ) break; if( zGlobList[0]=='\'' || zGlobList[0]=='"' ){ cTerm = zGlobList[0]; zGlobList++; }else{ cTerm = ','; } for(i=0; zGlobList[i] && zGlobList[i]!=cTerm; i++){} if( cTerm==',' ){ - while( i>0 && isspace(zGlobList[i-1]) ){ i--; } + while( i>0 && fossil_isspace(zGlobList[i-1]) ){ i--; } } blob_appendf(&expr, "%s%s GLOB '%.*q'", zSep, zVal, i, zGlobList); zSep = " OR "; if( cTerm!=',' && zGlobList[i] ) i++; zGlobList += i; @@ -259,10 +259,11 @@ Blob repo; Stmt q; int n; const char *zIgnoreFlag = find_option("ignore",0,1); int allFlag = find_option("dotfiles",0,0)!=0; + int outputManifest = db_get_boolean("manifest",0); db_must_be_within_tree(); db_multi_exec("CREATE TEMP TABLE sfile(x TEXT PRIMARY KEY)"); n = strlen(g.zLocalRoot); blob_init(&path, g.zLocalRoot, n-1); @@ -270,16 +271,18 @@ zIgnoreFlag = db_get("ignore-glob", 0); } vfile_scan(0, &path, blob_size(&path), allFlag); db_prepare(&q, "SELECT x FROM sfile" - " WHERE x NOT IN ('manifest','manifest.uuid','_FOSSIL_'," + " WHERE x NOT IN ('%s','%s','_FOSSIL_'," "'_FOSSIL_-journal','.fos','.fos-journal'," "'_FOSSIL_-wal','_FOSSIL_-shm','.fos-wal'," "'.fos-shm')" " AND NOT %s" " ORDER BY 1", + outputManifest ? "manifest" : "_FOSSIL_", + outputManifest ? "manifest.uuid" : "_FOSSIL_", glob_expr("x", zIgnoreFlag) ); if( file_tree_name(g.zRepositoryName, &repo, 0) ){ db_multi_exec("DELETE FROM sfile WHERE x=%B", &repo); } @@ -401,25 +404,25 @@ } if( zEditor==0 ){ zEditor = getenv("EDITOR"); } if( zEditor==0 ){ -#ifdef __MINGW32__ +#if defined(_WIN32) zEditor = "notepad"; #else zEditor = "ed"; #endif } zFile = db_text(0, "SELECT '%qci-comment-' || hex(randomblob(6)) || '.txt'", g.zLocalRoot); -#ifdef __MINGW32__ +#if defined(_WIN32) blob_add_cr(&text); #endif blob_write_to_file(&text, zFile); zCmd = mprintf("%s \"%s\"", zEditor, zFile); printf("%s\n", zCmd); - if( portable_system(zCmd) ){ + if( fossil_system(zCmd) ){ fossil_panic("editor aborted"); } blob_reset(&text); blob_read_from_file(&text, zFile); blob_remove_cr(&text); @@ -429,20 +432,20 @@ while( blob_line(&text, &line) ){ int i, n; char *z; n = blob_size(&line); z = blob_buffer(&line); - for(i=0; i<n && isspace(z[i]); i++){} + for(i=0; i<n && fossil_isspace(z[i]); i++){} if( i<n && z[i]=='#' ) continue; if( i<n || blob_size(pComment)>0 ){ blob_appendf(pComment, "%b", &line); } } blob_reset(&text); zComment = blob_str(pComment); i = strlen(zComment); - while( i>0 && isspace(zComment[i-1]) ){ i--; } + while( i>0 && fossil_isspace(zComment[i-1]) ){ i--; } blob_resize(pComment, i); } /* ** Populate the Global.aCommitFile[] based on the command line arguments @@ -461,11 +464,11 @@ void select_commit_files(void){ if( g.argc>2 ){ int ii; Blob b; blob_zero(&b); - g.aCommitFile = malloc(sizeof(int)*(g.argc-1)); + g.aCommitFile = fossil_malloc(sizeof(int)*(g.argc-1)); for(ii=2; ii<g.argc; ii++){ int iId; file_tree_name(g.argv[ii], &b, 1); iId = db_int(-1, "SELECT id FROM vfile WHERE pathname=%Q", blob_str(&b)); @@ -513,35 +516,201 @@ " WHERE datetime(mtime)>=%Q" " AND type='ci' AND objid=%d", zDate, rid ); if( b ){ - fossil_fatal("ancestor check-in [%.10s] (%s) is younger (clock skew?)", - zUuid, zDate); + fossil_fatal("ancestor check-in [%.10s] (%s) is not older (clock skew?)" + " Use -f to override.", zUuid, zDate); + } +#endif +} + +/* +** zDate should be a valid date string. Convert this string into the +** format YYYY-MM-DDTHH:MM:SS. If the string is not a valid date, +** print a fatal error and quit. +*/ +char *date_in_standard_format(const char *zInputDate){ + char *zDate = db_text(0, "SELECT datetime(%Q)", zInputDate); + if( zDate[0]==0 ){ + fossil_fatal("unrecognized date format (%s): use \"YYYY-MM-DD HH:MM:SS\"", + zInputDate); + } + assert( strlen(zDate)==19 ); + assert( zDate[10]==' ' ); + zDate[10] = 'T'; + return zDate; +} + +/* +** Create a manifest. +*/ +static void create_manifest( + Blob *pOut, /* Write the manifest here */ + const char *zBaselineUuid, /* UUID of baseline, or zero */ + Manifest *pBaseline, /* Make it a delta manifest if not zero */ + Blob *pComment, /* Check-in comment text */ + int vid, /* blob-id of the parent manifest */ + int verifyDate, /* Verify that child is younger */ + Blob *pCksum, /* Repository checksum. May be 0 */ + const char *zDateOvrd, /* Date override. If 0 then use 'now' */ + const char *zUserOvrd, /* User override. If 0 then use g.zLogin */ + const char *zBranch, /* Branch name. May be 0 */ + const char *zBgColor, /* Background color. May be 0 */ + int *pnFBcard /* Number of generated B- and F-cards */ +){ + char *zDate; /* Date of the check-in */ + char *zParentUuid; /* UUID of parent check-in */ + Blob filename; /* A single filename */ + int nBasename; /* Size of base filename */ + Stmt q; /* Query of files changed */ + Stmt q2; /* Query of merge parents */ + Blob mcksum; /* Manifest checksum */ + ManifestFile *pFile; /* File from the baseline */ + int nFBcard = 0; /* Number of B-cards and F-cards */ + + assert( pBaseline==0 || pBaseline->zBaseline==0 ); + assert( pBaseline==0 || zBaselineUuid!=0 ); + blob_zero(pOut); + zParentUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", vid); + if( pBaseline ){ + blob_appendf(pOut, "B %s\n", zBaselineUuid); + manifest_file_rewind(pBaseline); + pFile = manifest_file_next(pBaseline, 0); + nFBcard++; + }else{ + pFile = 0; } + blob_appendf(pOut, "C %F\n", blob_str(pComment)); + zDate = date_in_standard_format(zDateOvrd ? zDateOvrd : "now"); + blob_appendf(pOut, "D %s\n", zDate); + zDate[10] = ' '; + db_prepare(&q, + "SELECT pathname, uuid, origname, blob.rid, isexe" + " FROM vfile JOIN blob ON vfile.mrid=blob.rid" + " WHERE NOT deleted AND vfile.vid=%d" + " ORDER BY 1", vid); + blob_zero(&filename); + blob_appendf(&filename, "%s", g.zLocalRoot); + nBasename = blob_size(&filename); + while( db_step(&q)==SQLITE_ROW ){ + const char *zName = db_column_text(&q, 0); + const char *zUuid = db_column_text(&q, 1); + const char *zOrig = db_column_text(&q, 2); + int frid = db_column_int(&q, 3); + int isexe = db_column_int(&q, 4); + const char *zPerm; + int cmp; + blob_append(&filename, zName, -1); +#if !defined(_WIN32) + /* For unix, extract the "executable" permission bit directly from + ** the filesystem. On windows, the "executable" bit is retained + ** unchanged from the original. */ + isexe = file_isexe(blob_str(&filename)); #endif + if( isexe ){ + zPerm = " x"; + }else{ + zPerm = ""; + } + if( !g.markPrivate ) content_make_public(frid); + while( pFile && strcmp(pFile->zName,zName)<0 ){ + blob_appendf(pOut, "F %F\n", pFile->zName); + pFile = manifest_file_next(pBaseline, 0); + nFBcard++; + } + cmp = 1; + if( pFile==0 + || (cmp = strcmp(pFile->zName,zName))!=0 + || strcmp(pFile->zUuid, zUuid)!=0 + ){ + blob_resize(&filename, nBasename); + if( zOrig==0 || strcmp(zOrig,zName)==0 ){ + blob_appendf(pOut, "F %F %s%s\n", zName, zUuid, zPerm); + }else{ + if( zPerm[0]==0 ){ zPerm = " w"; } + blob_appendf(pOut, "F %F %s%s %F\n", zName, zUuid, zPerm, zOrig); + } + nFBcard++; + } + if( cmp==0 ) pFile = manifest_file_next(pBaseline,0); + } + blob_reset(&filename); + db_finalize(&q); + while( pFile ){ + blob_appendf(pOut, "F %F\n", pFile->zName); + pFile = manifest_file_next(pBaseline, 0); + nFBcard++; + } + blob_appendf(pOut, "P %s", zParentUuid); + if( verifyDate ) checkin_verify_younger(vid, zParentUuid, zDate); + free(zParentUuid); + db_prepare(&q2, "SELECT merge FROM vmerge WHERE id=:id"); + db_bind_int(&q2, ":id", 0); + while( db_step(&q2)==SQLITE_ROW ){ + char *zMergeUuid; + int mid = db_column_int(&q2, 0); + if( !g.markPrivate && content_is_private(mid) ) continue; + zMergeUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", mid); + if( zMergeUuid ){ + blob_appendf(pOut, " %s", zMergeUuid); + if( verifyDate ) checkin_verify_younger(mid, zMergeUuid, zDate); + free(zMergeUuid); + } + } + db_finalize(&q2); + free(zDate); + + blob_appendf(pOut, "\n"); + if( pCksum ) blob_appendf(pOut, "R %b\n", pCksum); + if( zBranch && zBranch[0] ){ + Stmt q; + if( zBgColor && zBgColor[0] ){ + blob_appendf(pOut, "T *bgcolor * %F\n", zBgColor); + } + blob_appendf(pOut, "T *branch * %F\n", zBranch); + blob_appendf(pOut, "T *sym-%F *\n", zBranch); + + /* Cancel all other symbolic tags */ + db_prepare(&q, + "SELECT tagname FROM tagxref, tag" + " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid" + " AND tagtype>0 AND tagname GLOB 'sym-*'" + " AND tagname!='sym-'||%Q" + " ORDER BY tagname", + vid, zBranch); + while( db_step(&q)==SQLITE_ROW ){ + const char *zTag = db_column_text(&q, 0); + blob_appendf(pOut, "T -%F *\n", zTag); + } + db_finalize(&q); + } + blob_appendf(pOut, "U %F\n", zUserOvrd ? zUserOvrd : g.zLogin); + md5sum_blob(pOut, &mcksum); + blob_appendf(pOut, "Z %b\n", &mcksum); + if( pnFBcard ) *pnFBcard = nFBcard; } + /* ** COMMAND: ci ** COMMAND: commit ** ** Usage: %fossil commit ?OPTIONS? ?FILE...? ** ** Create a new version containing all of the changes in the current ** checkout. You will be prompted to enter a check-in comment unless -** the comment has been specified on the command-line using "-m". -** The editor defined in the "editor" fossil option (see %fossil help set) -** will be used, or from the "VISUAL" or "EDITOR" environment variables -** (in that order) if no editor is set. -** -** You will be prompted for your GPG passphrase in order to sign the -** new manifest unless the "--nosign" option is used. All files that -** have changed will be committed unless some subset of files is -** specified on the command line. -** -** The --branch option followed by a branch name cases the new check-in +** the comment has been specified on the command-line using "-m" or a +** file containing the comment using -M. The editor defined in the +** "editor" fossil option (see %fossil help set) will be used, or from +** the "VISUAL" or "EDITOR" environment variables (in that order) if +** no editor is set. +** +** All files that have changed will be committed unless some subset of +** files is specified on the command line. +** +** The --branch option followed by a branch name causes the new check-in ** to be placed in the named branch. The --bgcolor option can be followed ** by a color name (ex: '#ffc0c0') to specify the background color of ** entries in the new branch when shown in the web timeline interface. ** ** A check-in is not permitted to fork unless the --force or -f @@ -551,44 +720,58 @@ ** Children of private check-ins are automatically private. ** ** Options: ** ** --comment|-m COMMENT-TEXT +** --message-file|-M COMMENT-FILE ** --branch NEW-BRANCH-NAME ** --bgcolor COLOR ** --nosign ** --force|-f ** --private +** --baseline +** --delta ** */ void commit_cmd(void){ - int rc; - int vid, nrid, nvid; - Blob comment; - const char *zComment; - Stmt q; - Stmt q2; - char *zUuid, *zDate; + int hasChanges; /* True if unsaved changes exist */ + int vid; /* blob-id of parent version */ + int nrid; /* blob-id of a modified file */ + int nvid; /* Blob-id of the new check-in */ + Blob comment; /* Check-in comment */ + const char *zComment; /* Check-in comment */ + Stmt q; /* Query to find files that have been modified */ + char *zUuid; /* UUID of the new check-in */ int noSign = 0; /* True to omit signing the manifest using GPG */ int isAMerge = 0; /* True if checking in a merge */ int forceFlag = 0; /* Force a fork */ + int forceDelta = 0; /* Force a delta-manifest */ + int forceBaseline = 0; /* Force a baseline-manifest */ char *zManifestFile; /* Name of the manifest file */ - int nBasename; /* Length of "g.zLocalRoot/" */ + int useCksum; /* True if checksums should be computed and verified */ + int outputManifest; /* True to output "manifest" and "manifest.uuid" */ + int testRun; /* True for a test run. Debugging only */ const char *zBranch; /* Create a new branch with this name */ const char *zBgColor; /* Set background color when branching */ const char *zDateOvrd; /* Override date string */ const char *zUserOvrd; /* Override user name */ const char *zComFile; /* Read commit message from this file */ - Blob filename; /* complete filename */ - Blob manifest; + Blob manifest; /* Manifest in baseline form */ Blob muuid; /* Manifest uuid */ - Blob mcksum; /* Self-checksum on the manifest */ Blob cksum1, cksum2; /* Before and after commit checksums */ Blob cksum1b; /* Checksum recorded in the manifest */ + int szD; /* Size of the delta manifest */ + int szB; /* Size of the baseline manifest */ url_proxy_options(); noSign = find_option("nosign",0,0)!=0; + forceDelta = find_option("delta",0,0)!=0; + forceBaseline = find_option("baseline",0,0)!=0; + if( forceDelta && forceBaseline ){ + fossil_fatal("cannot use --delta and --baseline together"); + } + testRun = find_option("test",0,0)!=0; zComment = find_option("comment","m",1); forceFlag = find_option("force", "f", 0)!=0; zBranch = find_option("branch","b",1); zBgColor = find_option("bgcolor",0,1); zComFile = find_option("message-file", "M", 1); @@ -600,11 +783,23 @@ zDateOvrd = find_option("date-override",0,1); zUserOvrd = find_option("user-override",0,1); db_must_be_within_tree(); noSign = db_get_boolean("omitsign", 0)|noSign; if( db_get_boolean("clearsign", 0)==0 ){ noSign = 1; } + useCksum = db_get_boolean("repo-cksum", 1); + outputManifest = db_get_boolean("manifest", 0); verify_all_options(); + + /* So that older versions of Fossil (that do not understand delta- + ** manifest) can continue to use this repository, do not create a new + ** delta-manifest unless this repository already contains one or more + ** delta-manifets, or unless the delta-manifest is explicitly requested + ** by the --delta option. + */ + if( !forceDelta && !db_get_boolean("seen-delta-manifest",0) ){ + forceBaseline = 1; + } /* Get the ID of the parent manifest artifact */ vid = db_lget_int("checkout", 0); if( content_is_private(vid) ){ g.markPrivate = 1; @@ -614,10 +809,22 @@ ** Autosync if autosync is enabled and this is not a private check-in. */ if( !g.markPrivate ){ autosync(AUTOSYNC_PULL); } + + /* Require confirmation to continue with the check-in if there is + ** clock skew + */ + if( g.clockSkewSeen ){ + Blob ans; + blob_zero(&ans); + prompt_user("continue in spite of time skew (y/N)? ", &ans); + if( blob_str(&ans)[0]!='y' ){ + fossil_exit(1); + } + } /* There are two ways this command may be executed. If there are ** no arguments following the word "commit", then all modified files ** in the checked out directory are committed. If one or more arguments ** follows "commit", then only those files are committed. @@ -639,15 +846,15 @@ */ if( !db_exists("SELECT 1 FROM user WHERE login=%Q", g.zLogin) ){ fossil_fatal("no such user: %s", g.zLogin); } + hasChanges = unsaved_changes(); db_begin_transaction(); db_record_repository_filename(0); - rc = unsaved_changes(); - if( rc==0 && !isAMerge && !forceFlag ){ - fossil_panic("nothing has changed"); + if( hasChanges==0 && !isAMerge && !forceFlag ){ + fossil_fatal("nothing has changed"); } /* If one or more files that were named on the command line have not ** been modified, bail out now. */ @@ -678,11 +885,11 @@ " WHERE tagid=%d AND rid=%d AND tagtype>0", TAG_CLOSED, vid) ){ fossil_fatal("cannot commit against a closed leaf"); } - vfile_aggregate_checksum_disk(vid, &cksum1); + if( useCksum ) vfile_aggregate_checksum_disk(vid, &cksum1); if( zComment ){ blob_zero(&comment); blob_append(&comment, zComment, -1); }else if( zComFile ){ blob_zero(&comment); @@ -733,131 +940,109 @@ db_multi_exec("UPDATE vfile SET mrid=%d, rid=%d WHERE id=%d", nrid,nrid,id); db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", nrid); } db_finalize(&q); - /* Create the manifest */ - blob_zero(&manifest); + /* Create the new manifest */ if( blob_size(&comment)==0 ){ blob_append(&comment, "(no comment)", -1); } - blob_appendf(&manifest, "C %F\n", blob_str(&comment)); - zDate = db_text(0, "SELECT datetime('%q')", zDateOvrd ? zDateOvrd : "now"); - zDate[10] = 'T'; - blob_appendf(&manifest, "D %s\n", zDate); - zDate[10] = ' '; - db_prepare(&q, - "SELECT pathname, uuid, origname, blob.rid, isexe" - " FROM vfile JOIN blob ON vfile.mrid=blob.rid" - " WHERE NOT deleted AND vfile.vid=%d" - " ORDER BY 1", vid); - blob_zero(&filename); - blob_appendf(&filename, "%s", g.zLocalRoot); - nBasename = blob_size(&filename); - while( db_step(&q)==SQLITE_ROW ){ - const char *zName = db_column_text(&q, 0); - const char *zUuid = db_column_text(&q, 1); - const char *zOrig = db_column_text(&q, 2); - int frid = db_column_int(&q, 3); - int isexe = db_column_int(&q, 4); - const char *zPerm; - blob_append(&filename, zName, -1); -#ifndef __MINGW32__ - /* For unix, extract the "executable" permission bit directly from - ** the filesystem. On windows, the "executable" bit is retained - ** unchanged from the original. */ - isexe = file_isexe(blob_str(&filename)); -#endif - if( isexe ){ - zPerm = " x"; - }else{ - zPerm = ""; - } - blob_resize(&filename, nBasename); - if( zOrig==0 || strcmp(zOrig,zName)==0 ){ - blob_appendf(&manifest, "F %F %s%s\n", zName, zUuid, zPerm); - }else{ - if( zPerm[0]==0 ){ zPerm = " w"; } - blob_appendf(&manifest, "F %F %s%s %F\n", zName, zUuid, zPerm, zOrig); - } - if( !g.markPrivate ) content_make_public(frid); - } - blob_reset(&filename); - db_finalize(&q); - zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", vid); - blob_appendf(&manifest, "P %s", zUuid); - checkin_verify_younger(vid, zUuid, zDate); - - db_prepare(&q2, "SELECT merge FROM vmerge WHERE id=:id"); - db_bind_int(&q2, ":id", 0); - while( db_step(&q2)==SQLITE_ROW ){ - int mid = db_column_int(&q2, 0); - if( !g.markPrivate && content_is_private(mid) ) continue; - zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", mid); - if( zUuid ){ - blob_appendf(&manifest, " %s", zUuid); - checkin_verify_younger(mid, zUuid, zDate); - free(zUuid); - } - } - db_reset(&q2); - - blob_appendf(&manifest, "\n"); - blob_appendf(&manifest, "R %b\n", &cksum1); - if( zBranch && zBranch[0] ){ - Stmt q; - if( zBgColor && zBgColor[0] ){ - blob_appendf(&manifest, "T *bgcolor * %F\n", zBgColor); - } - blob_appendf(&manifest, "T *branch * %F\n", zBranch); - blob_appendf(&manifest, "T *sym-%F *\n", zBranch); - - /* Cancel all other symbolic tags */ - db_prepare(&q, - "SELECT tagname FROM tagxref, tag" - " WHERE tagxref.rid=%d AND tagxref.tagid=tag.tagid" - " AND tagtype>0 AND tagname GLOB 'sym-*'" - " AND tagname!='sym-'||%Q" - " ORDER BY tagname", - vid, zBranch); - while( db_step(&q)==SQLITE_ROW ){ - const char *zTag = db_column_text(&q, 0); - blob_appendf(&manifest, "T -%F *\n", zTag); - } - db_finalize(&q); - } - blob_appendf(&manifest, "U %F\n", zUserOvrd ? zUserOvrd : g.zLogin); - md5sum_blob(&manifest, &mcksum); - blob_appendf(&manifest, "Z %b\n", &mcksum); - zManifestFile = mprintf("%smanifest", g.zLocalRoot); + if( forceDelta ){ + blob_zero(&manifest); + }else{ + create_manifest(&manifest, 0, 0, &comment, vid, + !forceFlag, useCksum ? &cksum1 : 0, + zDateOvrd, zUserOvrd, zBranch, zBgColor, &szB); + } + + /* See if a delta-manifest would be more appropriate */ + if( !forceBaseline ){ + const char *zBaselineUuid; + Manifest *pParent; + Manifest *pBaseline; + pParent = manifest_get(vid, CFTYPE_MANIFEST); + if( pParent && pParent->zBaseline ){ + zBaselineUuid = pParent->zBaseline; + pBaseline = manifest_get_by_name(zBaselineUuid, 0); + }else{ + zBaselineUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", vid); + pBaseline = pParent; + } + if( pBaseline ){ + Blob delta; + create_manifest(&delta, zBaselineUuid, pBaseline, &comment, vid, + !forceFlag, useCksum ? &cksum1 : 0, + zDateOvrd, zUserOvrd, zBranch, zBgColor, &szD); + /* + ** At this point, two manifests have been constructed, either of + ** which would work for this checkin. The first manifest (held + ** in the "manifest" variable) is a baseline manifest and the second + ** (held in variable named "delta") is a delta manifest. The + ** question now is: which manifest should we use? + ** + ** Let B be the number of F-cards in the baseline manifest and + ** let D be the number of F-cards in the delta manifest, plus one for + ** the B-card. (B is held in the szB variable and D is held in the + ** szD variable.) Assume that all delta manifests adds X new F-cards. + ** Then to minimize the total number of F- and B-cards in the repository, + ** we should use the delta manifest if and only if: + ** + ** D*D < B*X - X*X + ** + ** X is an unknown here, but for most repositories, we will not be + ** far wrong if we assume X=3. + */ + if( forceDelta || (szD*szD)<(szB*3-9) ){ + blob_reset(&manifest); + manifest = delta; + }else{ + blob_reset(&delta); + } + }else if( forceDelta ){ + fossil_panic("unable to find a baseline-manifest for the delta"); + } + } if( !noSign && !g.markPrivate && clearsign(&manifest, &manifest) ){ Blob ans; blob_zero(&ans); prompt_user("unable to sign manifest. continue (y/N)? ", &ans); if( blob_str(&ans)[0]!='y' ){ fossil_exit(1); } } - blob_write_to_file(&manifest, zManifestFile); - blob_reset(&manifest); - blob_read_from_file(&manifest, zManifestFile); - free(zManifestFile); + + /* If the --test option is specified, output the manifest file + ** and rollback the transaction. + */ + if( testRun ){ + blob_write_to_file(&manifest, ""); + } + + if( outputManifest ){ + zManifestFile = mprintf("%smanifest", g.zLocalRoot); + blob_write_to_file(&manifest, zManifestFile); + blob_reset(&manifest); + blob_read_from_file(&manifest, zManifestFile); + free(zManifestFile); + } nvid = content_put(&manifest, 0, 0); if( nvid==0 ){ fossil_panic("trouble committing manifest: %s", g.zErrMsg); } db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", nvid); manifest_crosslink(nvid, &manifest); content_deltify(vid, nvid, 0); zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", nvid); printf("New_Version: %s\n", zUuid); - zManifestFile = mprintf("%smanifest.uuid", g.zLocalRoot); - blob_zero(&muuid); - blob_appendf(&muuid, "%s\n", zUuid); - blob_write_to_file(&muuid, zManifestFile); - free(zManifestFile); - blob_reset(&muuid); + if( outputManifest ){ + zManifestFile = mprintf("%smanifest.uuid", g.zLocalRoot); + blob_zero(&muuid); + blob_appendf(&muuid, "%s\n", zUuid); + blob_write_to_file(&muuid, zManifestFile); + free(zManifestFile); + blob_reset(&muuid); + } /* Update the vfile and vmerge tables */ db_multi_exec( "DELETE FROM vfile WHERE (vid!=%d OR deleted) AND file_is_selected(id);" @@ -867,45 +1052,51 @@ " WHERE file_is_selected(id);" , vid, nvid ); db_lset_int("checkout", nvid); - /* Verify that the repository checksum matches the expected checksum - ** calculated before the checkin started (and stored as the R record - ** of the manifest file). - */ - vfile_aggregate_checksum_repository(nvid, &cksum2); - if( blob_compare(&cksum1, &cksum2) ){ - fossil_panic("tree checksum does not match repository after commit"); - } - - /* Verify that the manifest checksum matches the expected checksum */ - vfile_aggregate_checksum_manifest(nvid, &cksum2, &cksum1b); - if( blob_compare(&cksum1, &cksum1b) ){ - fossil_panic("manifest checksum does not agree with manifest: " - "%b versus %b", &cksum1, &cksum1b); - } - if( blob_compare(&cksum1, &cksum2) ){ - fossil_panic("tree checksum does not match manifest after commit: " - "%b versus %b", &cksum1, &cksum2); - } - - /* Verify that the commit did not modify any disk images. */ - vfile_aggregate_checksum_disk(nvid, &cksum2); - if( blob_compare(&cksum1, &cksum2) ){ - fossil_panic("tree checksums before and after commit do not match"); + if( useCksum ){ + /* Verify that the repository checksum matches the expected checksum + ** calculated before the checkin started (and stored as the R record + ** of the manifest file). + */ + vfile_aggregate_checksum_repository(nvid, &cksum2); + if( blob_compare(&cksum1, &cksum2) ){ + fossil_panic("tree checksum does not match repository after commit"); + } + + /* Verify that the manifest checksum matches the expected checksum */ + vfile_aggregate_checksum_manifest(nvid, &cksum2, &cksum1b); + if( blob_compare(&cksum1, &cksum1b) ){ + fossil_panic("manifest checksum does not agree with manifest: " + "%b versus %b", &cksum1, &cksum1b); + } + if( blob_compare(&cksum1, &cksum2) ){ + fossil_panic("tree checksum does not match manifest after commit: " + "%b versus %b", &cksum1, &cksum2); + } + + /* Verify that the commit did not modify any disk images. */ + vfile_aggregate_checksum_disk(nvid, &cksum2); + if( blob_compare(&cksum1, &cksum2) ){ + fossil_panic("tree checksums before and after commit do not match"); + } } /* Clear the undo/redo stack */ undo_reset(); /* Commit */ db_multi_exec("DELETE FROM vvar WHERE name='ci-comment'"); + if( testRun ){ + db_end_transaction(1); + exit(1); + } db_end_transaction(0); if( !g.markPrivate ){ autosync(AUTOSYNC_PUSH); } if( count_nonbranch_children(vid)>1 ){ printf("**** warning: a fork has occurred *****\n"); } } Index: src/checkout.c ================================================================== --- src/checkout.c +++ src/checkout.c @@ -78,67 +78,99 @@ /* ** Load a vfile from a record ID. */ void load_vfile_from_rid(int vid){ - Blob manifest; - if( db_exists("SELECT 1 FROM vfile WHERE vid=%d", vid) ){ return; } - content_get(vid, &manifest); - vfile_build(vid, &manifest); - blob_reset(&manifest); + vfile_build(vid); } /* ** Set or clear the vfile.isexe flag for a file. */ static void set_or_clear_isexe(const char *zFilename, int vid, int onoff){ - db_multi_exec("UPDATE vfile SET isexe=%d WHERE vid=%d and pathname=%Q", - onoff, vid, zFilename); + static Stmt s; + db_static_prepare(&s, + "UPDATE vfile SET isexe=:isexe" + " WHERE vid=:vid AND pathname=:path AND isexe!=:isexe" + ); + db_bind_int(&s, ":isexe", onoff); + db_bind_int(&s, ":vid", vid); + db_bind_text(&s, ":path", zFilename); + db_step(&s); + db_reset(&s); +} + +/* +** Set or clear the execute permission bit (as appropriate) for all +** files in the current check-out. +*/ +void checkout_set_all_exe(int vid){ + Blob filename; + int baseLen; + Manifest *pManifest; + ManifestFile *pFile; + + /* Check the EXE permission status of all files + */ + pManifest = manifest_get(vid, CFTYPE_MANIFEST); + if( pManifest==0 ) return; + blob_zero(&filename); + blob_appendf(&filename, "%s/", g.zLocalRoot); + baseLen = blob_size(&filename); + manifest_file_rewind(pManifest); + while( (pFile = manifest_file_next(pManifest, 0))!=0 ){ + int isExe; + blob_append(&filename, pFile->zName, -1); + isExe = pFile->zPerm && strstr(pFile->zPerm, "x"); + file_setexe(blob_str(&filename), isExe); + set_or_clear_isexe(pFile->zName, vid, isExe); + blob_resize(&filename, baseLen); + } + blob_reset(&filename); + manifest_destroy(pManifest); } + /* -** Read the manifest file given by vid out of the repository -** and store it in the root of the local check-out. +** If the "manifest" setting is true, then automatically generate +** files named "manifest" and "manifest.uuid" containing, respectively, +** the text of the manifest and the artifact ID of the manifest. */ void manifest_to_disk(int vid){ char *zManFile; Blob manifest; Blob hash; - Blob filename; - int baseLen; - int i; - Manifest m; - - blob_zero(&manifest); - zManFile = mprintf("%smanifest", g.zLocalRoot); - content_get(vid, &manifest); - blob_write_to_file(&manifest, zManFile); - free(zManFile); - blob_zero(&hash); - sha1sum_blob(&manifest, &hash); - zManFile = mprintf("%smanifest.uuid", g.zLocalRoot); - blob_append(&hash, "\n", 1); - blob_write_to_file(&hash, zManFile); - free(zManFile); - blob_reset(&hash); - manifest_parse(&m, &manifest); - blob_zero(&filename); - blob_appendf(&filename, "%s/", g.zLocalRoot); - baseLen = blob_size(&filename); - for(i=0; i<m.nFile; i++){ - int isExe; - blob_append(&filename, m.aFile[i].zName, -1); - isExe = m.aFile[i].zPerm && strstr(m.aFile[i].zPerm, "x"); - file_setexe(blob_str(&filename), isExe); - set_or_clear_isexe(m.aFile[i].zName, vid, isExe); - blob_resize(&filename, baseLen); - } - blob_reset(&filename); - manifest_clear(&m); + + if( db_get_boolean("manifest",0) ){ + blob_zero(&manifest); + content_get(vid, &manifest); + zManFile = mprintf("%smanifest", g.zLocalRoot); + blob_write_to_file(&manifest, zManFile); + free(zManFile); + blob_zero(&hash); + sha1sum_blob(&manifest, &hash); + zManFile = mprintf("%smanifest.uuid", g.zLocalRoot); + blob_append(&hash, "\n", 1); + blob_write_to_file(&hash, zManFile); + free(zManFile); + blob_reset(&hash); + }else{ + if( !db_exists("SELECT 1 FROM vfile WHERE pathname='manifest'") ){ + zManFile = mprintf("%smanifest", g.zLocalRoot); + unlink(zManFile); + free(zManFile); + } + if( !db_exists("SELECT 1 FROM vfile WHERE pathname='manifest.uuid'") ){ + zManFile = mprintf("%smanifest.uuid", g.zLocalRoot); + unlink(zManFile); + free(zManFile); + } + } + } /* ** COMMAND: checkout ** COMMAND: co @@ -208,10 +240,11 @@ } db_multi_exec("DELETE FROM vfile WHERE vid!=%d", vid); if( !keepFlag ){ vfile_to_disk(vid, 0, 1, promptFlag); } + checkout_set_all_exe(vid); manifest_to_disk(vid); db_lset_int("checkout", vid); undo_reset(); db_multi_exec("DELETE FROM vmerge"); if( !keepFlag ){ @@ -218,11 +251,11 @@ vfile_aggregate_checksum_manifest(vid, &cksum1, &cksum1b); vfile_aggregate_checksum_disk(vid, &cksum2); if( blob_compare(&cksum1, &cksum2) ){ printf("WARNING: manifest checksum does not agree with disk\n"); } - if( blob_compare(&cksum1, &cksum1b) ){ + if( blob_size(&cksum1b) && blob_compare(&cksum1, &cksum1b) ){ printf("WARNING: manifest checksum does not agree with manifest\n"); } } db_end_transaction(0); } Index: src/clearsign.c ================================================================== --- src/clearsign.c +++ src/clearsign.c @@ -39,11 +39,11 @@ zRand = db_text(0, "SELECT hex(randomblob(10))"); zOut = mprintf("out-%s", zRand); zIn = mprintf("in-%z", zRand); blob_write_to_file(pIn, zOut); zCmd = mprintf("%s %s %s", zBase, zIn, zOut); - rc = portable_system(zCmd); + rc = fossil_system(zCmd); free(zCmd); if( rc==0 ){ if( pOut==pIn ){ blob_reset(pIn); } Index: src/comformat.c ================================================================== --- src/comformat.c +++ src/comformat.c @@ -38,29 +38,29 @@ int doIndent = 0; char zBuf[400]; int lineCnt = 0; for(;;){ - while( isspace(zText[0]) ){ zText++; } + while( fossil_isspace(zText[0]) ){ zText++; } if( zText[0]==0 ){ if( doIndent==0 ){ printf("\n"); lineCnt = 1; } return lineCnt; } for(sk=si=i=k=0; zText[i] && k<tlen; i++){ char c = zText[i]; - if( isspace(c) ){ + if( fossil_isspace(c) ){ si = i; sk = k; if( k==0 || zBuf[k-1]!=' ' ){ zBuf[k++] = ' '; } }else{ zBuf[k] = c; - if( c=='-' && k>0 && isalpha(zBuf[k-1]) ){ + if( c=='-' && k>0 && fossil_isalpha(zBuf[k-1]) ){ si = i+1; sk = k+1; } k++; } Index: src/config.h ================================================================== --- src/config.h +++ src/config.h @@ -16,87 +16,116 @@ ******************************************************************************* ** ** A common header file used by all modules. */ +/* The following macros are necessary for large-file support under +** some linux distributions, and possibly other unixes as well. +*/ +#define _LARGE_FILE 1 +#ifndef _FILE_OFFSET_BITS +# define _FILE_OFFSET_BITS 64 +#endif +#define _LARGEFILE_SOURCE 1 + +#ifndef _RC_COMPILE_ + /* ** System header files used by all modules */ #include <unistd.h> #include <stdio.h> #include <stdlib.h> -#include <ctype.h> +/* #include <ctype.h> // do not use - causes problems */ #include <string.h> #include <stdarg.h> #include <assert.h> -#ifdef __MINGW32__ -# include <windows.h> + +#endif + +#if defined( __MINGW32__) || defined(__DMC__) || defined(_MSC_VER) || defined(__POCC__) +# if defined(__DMC__) || defined(_MSC_VER) || defined(__POCC__) + typedef int socklen_t; +# endif +# ifndef _WIN32 +# define _WIN32 +# endif #else +# include <sys/types.h> +# include <signal.h> # include <pwd.h> #endif +/* +** Define the compiler variant, used to compile the project +*/ +#if !defined(COMPILER_NAME) +# if defined(__DMC__) +# define COMPILER_NAME "dmc" +# elif defined(__POCC__) +# if defined(_M_X64) +# define COMPILER_NAME "pellesc64" +# else +# define COMPILER_NAME "pellesc32" +# endif +# elif defined(_MSC_VER) +# define COMPILER_NAME "msc" +# elif defined(__MINGW32__) +# define COMPILER_NAME "mingw32" +# elif defined(_WIN32) +# define COMPILER_NAME "win32" +# elif defined(__GNUC__) +# define COMPILER_NAME "gcc-" __VERSION__ +# else +# define COMPILER_NAME "unknown" +# endif +#endif + +#ifndef _RC_COMPILE_ + #include "sqlite3.h" /* ** Typedef for a 64-bit integer */ -typedef sqlite_int64 i64; -typedef sqlite_uint64 u64; +typedef sqlite3_int64 i64; +typedef sqlite3_uint64 u64; /* ** Unsigned character type */ typedef unsigned char u8; -/* -** Standard colors. These colors can also be changed using a stylesheet. -*/ - -/* A blue border and background. Used for the title bar and for dates -** in a timeline. -*/ -#define BORDER1 "#a0b5f4" /* Stylesheet class: border1 */ -#define BG1 "#d0d9f4" /* Stylesheet class: bkgnd1 */ - -/* A red border and background. Use for releases in the timeline. -*/ -#define BORDER2 "#ec9898" /* Stylesheet class: border2 */ -#define BG2 "#f7c0c0" /* Stylesheet class: bkgnd2 */ - -/* A gray background. Used for column headers in the Wiki Table of Contents -** and to highlight ticket properties. -*/ -#define BG3 "#d0d0d0" /* Stylesheet class: bkgnd3 */ - -/* A light-gray background. Used for title bar, menus, and rlog alternation -*/ -#define BG4 "#f0f0f0" /* Stylesheet class: bkgnd4 */ - -/* A deeper gray background. Used for branches -*/ -#define BG5 "#dddddd" /* Stylesheet class: bkgnd5 */ - -/* Default HTML page header */ -#define HEADER "<html>\n" \ - "<head>\n" \ - "<link rel=\"alternate\" type=\"application/rss+xml\"\n" \ - " title=\"%N Timeline Feed\" href=\"%B/timeline.rss\">\n" \ - "<title>%N: %T\n\n" \ - "" - -/* Default HTML page footer */ -#define FOOTER "\n" \ - "\n" - /* In the timeline, check-in messages are truncated at the first space ** that is more than MX_CKIN_MSG from the beginning, or at the first ** paragraph break that is more than MN_CKIN_MSG from the beginning. */ #define MN_CKIN_MSG 100 #define MX_CKIN_MSG 300 + +/* +** The following macros are used to cast pointers to integers and +** integers to pointers. The way you do this varies from one compiler +** to the next, so we have developed the following set of #if statements +** to generate appropriate macros for a wide range of compilers. +** +** The correct "ANSI" way to do this is to use the intptr_t type. +** Unfortunately, that typedef is not available on all compilers, or +** if it is available, it requires an #include of specific headers +** that vary from one machine to the next. +*/ +#if defined(__PTRDIFF_TYPE__) /* This case should work for GCC */ +# define FOSSIL_INT_TO_PTR(X) ((void*)(__PTRDIFF_TYPE__)(X)) +# define FOSSIL_PTR_TO_INT(X) ((int)(__PTRDIFF_TYPE__)(X)) +#elif !defined(__GNUC__) /* Works for compilers other than LLVM */ +# define FOSSIL_INT_TO_PTR(X) ((void*)&((char*)0)[X]) +# define FOSSIL_PTR_TO_INT(X) ((int)(((char*)X)-(char*)0)) +#else /* Generates a warning - but it always works */ +# define FOSSIL_INT_TO_PTR(X) ((void*)(X)) +# define FOSSIL_PTR_TO_INT(X) ((int)(X)) +#endif + /* Unset the following to disable internationalization code. */ #ifndef FOSSIL_I18N # define FOSSIL_I18N 1 #endif @@ -107,5 +136,7 @@ #endif #ifndef CODESET # undef FOSSIL_I18N # define FOSSIL_I18N 0 #endif + +#endif /* _RC_COMPILE_ */ Index: src/configure.c ================================================================== --- src/configure.c +++ src/configure.c @@ -74,10 +74,11 @@ { "footer", CONFIGSET_SKIN }, { "logo-mimetype", CONFIGSET_SKIN }, { "logo-image", CONFIGSET_SKIN }, { "project-name", CONFIGSET_PROJ }, { "project-description", CONFIGSET_PROJ }, + { "manifest", CONFIGSET_PROJ }, { "index-page", CONFIGSET_SKIN }, { "timeline-block-markup", CONFIGSET_SKIN }, { "timeline-max-comment", CONFIGSET_SKIN }, { "ticket-table", CONFIGSET_TKT }, { "ticket-common", CONFIGSET_TKT }, @@ -454,15 +455,16 @@ }else{ zServer = db_get("last-sync-url", 0); if( zServer==0 ){ fossil_fatal("no server specified"); } - zPw = db_get("last-sync-pw", 0); + zPw = unobscure(db_get("last-sync-pw", 0)); } url_parse(zServer); if( g.urlPasswd==0 && zPw ) g.urlPasswd = mprintf("%s", zPw); user_select(); + url_enable_proxy("via proxy: "); if( strncmp(zMethod, "push", n)==0 ){ client_sync(0,0,0,0,mask); }else{ client_sync(0,0,0,mask,0); } Index: src/content.c ================================================================== --- src/content.c +++ src/content.c @@ -19,33 +19,25 @@ */ #include "config.h" #include "content.h" #include -/* -** Macros for debugging -*/ -#if 0 -# define CONTENT_TRACE(X) printf X; -#else -# define CONTENT_TRACE(X) -#endif - /* ** The artifact retrival cache */ -#define MX_CACHE_CNT 50 /* Maximum number of positive cache entries */ -#define EXPELL_INTERVAL 5 /* How often to expell from a full cache */ static struct { - int n; /* Current number of positive cache entries */ + i64 szTotal; /* Total size of all entries in the cache */ + int n; /* Current number of eache entries */ + int nAlloc; /* Number of slots allocated in a[] */ int nextAge; /* Age counter for implementing LRU */ int skipCnt; /* Used to limit entries expelled from cache */ - struct { /* One instance of this for each cache entry */ + struct cacheLine { /* One instance of this for each cache entry */ int rid; /* Artifact id */ int age; /* Age. Newer is larger */ Blob content; /* Content of the artifact */ - } a[MX_CACHE_CNT]; /* The positive cache */ + } *a; /* The positive cache */ + Bag inCache; /* Set of artifacts currently in cache */ /* ** The missing artifact cache. ** ** Artifacts whose record ID are in missingCache cannot be retrieved @@ -56,10 +48,53 @@ */ Bag missing; /* Cache of artifacts that are incomplete */ Bag available; /* Cache of artifacts that are complete */ } contentCache; +/* +** Remove the oldest element from the content cache +*/ +static void content_cache_expire_oldest(void){ + int i; + int mnAge = contentCache.nextAge; + int mn = -1; + for(i=0; i=0 ){ + bag_remove(&contentCache.inCache, contentCache.a[mn].rid); + contentCache.szTotal -= blob_size(&contentCache.a[mn].content); + blob_reset(&contentCache.a[mn].content); + contentCache.n--; + contentCache.a[mn] = contentCache.a[contentCache.n]; + } +} + +/* +** Add an entry to the content cache +*/ +void content_cache_insert(int rid, Blob *pBlob){ + struct cacheLine *p; + if( contentCache.n>500 || contentCache.szTotal>50000000 ){ + content_cache_expire_oldest(); + } + if( contentCache.n>=contentCache.nAlloc ){ + contentCache.nAlloc = contentCache.nAlloc*2 + 10; + contentCache.a = fossil_realloc(contentCache.a, + contentCache.nAlloc*sizeof(contentCache.a[0])); + } + p = &contentCache.a[contentCache.n++]; + p->rid = rid; + p->age = contentCache.nextAge++; + contentCache.szTotal += blob_size(pBlob); + p->content = *pBlob; + blob_zero(pBlob); + bag_insert(&contentCache.inCache, rid); +} /* ** Clear the content cache. */ void content_clear_cache(void){ @@ -67,11 +102,13 @@ for(i=0; i0 ){ + n++; + if( n>=nAlloc ){ + nAlloc = nAlloc*2 + 10; + a = fossil_realloc(a, nAlloc*sizeof(a[0])); + } + a[n] = nextRid; + } + mx = n; + rc = content_get(a[n], pBlob); + n--; + while( rc && n>=0 ){ + rc = content_of_blob(a[n], &delta); + if( rc ){ + blob_delta_apply(pBlob, &delta, &next); blob_reset(&delta); - rc = 1; - } - - /* Save the srcid artifact in the cache */ - if( contentCache.n=0 ){ - contentCache.a[i].content = src; - contentCache.a[i].age = contentCache.nextAge++; - contentCache.a[i].rid = srcid; - CONTENT_TRACE(("%*sadd %d to cache\n", - bag_count(&inProcess), "", srcid)) - }else{ - blob_reset(&src); - } - } - bag_remove(&inProcess, srcid); - }else{ - /* No delta required. Read content directly from the database */ - if( content_of_blob(rid, pBlob) ){ - rc = 1; - } + if( (mx-n)%8==0 ){ + content_cache_insert(a[n+1], pBlob); + }else{ + blob_reset(pBlob); + } + *pBlob = next; + } + n--; + } + free(a); + if( !rc ) blob_reset(pBlob); } if( rc==0 ){ bag_insert(&contentCache.missing, rid); }else{ bag_insert(&contentCache.available, rid); @@ -276,25 +281,29 @@ } return rc; } /* -** COMMAND: artifact +** COMMAND: artifact ** -** Usage: %fossil artifact ARTIFACT-ID ?OUTPUT-FILENAME? +** Usage: %fossil artifact ARTIFACT-ID ?OUTPUT-FILENAME? ?OPTIONS? ** ** Extract an artifact by its SHA1 hash and write the results on ** standard output, or if the optional 4th argument is given, in ** the named output file. +** +** Options: +** +** -R|--repository FILE Extract artifacts from repository FILE */ void artifact_cmd(void){ int rid; Blob content; const char *zFile; - if( g.argc!=4 && g.argc!=3 ) usage("RECORDID ?FILENAME?"); + db_find_and_open_repository(1); + if( g.argc!=4 && g.argc!=3 ) usage("ARTIFACT-ID ?FILENAME? ?OPTIONS?"); zFile = g.argc==4 ? g.argv[3] : "-"; - db_must_be_within_tree(); rid = name_to_rid(g.argv[2]); content_get(rid, &content); blob_write_to_file(&content, zFile); } @@ -316,39 +325,99 @@ db_blob(&content, "SELECT content FROM blob WHERE rid=%d", rid); blob_uncompress(&content, &content); blob_write_to_file(&content, zFile); } +/* +** The following flag is set to disable the automatic calls to +** manifest_crosslink() when a record is dephantomized. This +** flag can be set (for example) when doing a clone when we know +** that rebuild will be run over all records at the conclusion +** of the operation. +*/ +static int ignoreDephantomizations = 0; + /* ** When a record is converted from a phantom to a real record, ** if that record has other records that are derived by delta, ** then call manifest_crosslink() on those other records. +** +** If the formerly phantom record or any of the other records +** derived by delta from the former phantom are a baseline manifest, +** then also invoke manifest_crosslink() on the delta-manifests +** associated with that baseline. +** +** Tail recursion is used to minimize stack depth. */ void after_dephantomize(int rid, int linkFlag){ Stmt q; - int prevTid = 0; - - /* The prevTid variable is used to delay invoking this routine - ** recursively, if possible, until after the query has finalized, - ** in order to avoid having an excessive number of prepared statements. - ** This is most effective in the common case where the query returns - ** just one row. - */ - db_prepare(&q, "SELECT rid FROM delta WHERE srcid=%d", rid); - while( db_step(&q)==SQLITE_ROW ){ - int tid = db_column_int(&q, 0); - if( prevTid ) after_dephantomize(prevTid, 1); - prevTid = tid; - } - db_finalize(&q); - if( prevTid ) after_dephantomize(prevTid, 1); - if( linkFlag ){ - Blob content; - content_get(rid, &content); - manifest_crosslink(rid, &content); - blob_reset(&content); - } + int nChildAlloc = 0; + int *aChild = 0; + Blob content; + + if( ignoreDephantomizations ) return; + while( rid ){ + int nChildUsed = 0; + int i; + + /* Parse the object rid itself */ + if( linkFlag ){ + content_get(rid, &content); + manifest_crosslink(rid, &content); + blob_reset(&content); + } + + /* Parse all delta-manifests that depend on baseline-manifest rid */ + db_prepare(&q, "SELECT rid FROM orphan WHERE baseline=%d", rid); + while( db_step(&q)==SQLITE_ROW ){ + int child = db_column_int(&q, 0); + if( nChildUsed>=nChildAlloc ){ + nChildAlloc = nChildAlloc*2 + 10; + aChild = fossil_realloc(aChild, nChildAlloc*sizeof(aChild)); + } + aChild[nChildUsed++] = child; + } + db_finalize(&q); + for(i=0; i=nChildAlloc ){ + nChildAlloc = nChildAlloc*2 + 10; + aChild = fossil_realloc(aChild, nChildAlloc*sizeof(aChild)); + } + aChild[nChildUsed++] = child; + } + db_finalize(&q); + for(i=1; i0 ? aChild[0] : 0; + linkFlag = 1; + } + free(aChild); +} + +/* +** Turn dephantomization processing on or off. +*/ +void content_enable_dephantomize(int onoff){ + ignoreDephantomizations = !onoff; } /* ** Write content into the database. Return the record ID. If the ** content is already in the database, just return the record ID. Index: src/db.c ================================================================== --- src/db.c +++ src/db.c @@ -27,10 +27,13 @@ ** (3) A local checkout database named "_FOSSIL_" or ".fos" ** and located at the root of the local copy of the source tree. ** */ #include "config.h" +#if ! defined(_WIN32) +# include +#endif #include #include #include #include #include "db.h" @@ -134,16 +137,16 @@ if( busy || g.db==0 ) return; busy = 1; undo_rollback(); if( nBegin ){ sqlite3_exec(g.db, "ROLLBACK", 0, 0, 0); + nBegin = 0; if( isNewRepo ){ db_close(); unlink(g.zRepositoryName); } } - nBegin = 0; busy = 0; } /* ** Install a commit hook. Hooks are installed in sequence order. @@ -534,11 +537,11 @@ } db_finalize(&s); return z; } -#ifdef __MINGW32__ +#if defined(_WIN32) extern char *sqlite3_win32_mbcs_to_utf8(const char*); #endif /* ** Initialize a new database file with the given schema. If anything @@ -552,11 +555,11 @@ sqlite3 *db; int rc; const char *zSql; va_list ap; -#ifdef __MINGW32__ +#if defined(_WIN32) zFileName = sqlite3_win32_mbcs_to_utf8(zFileName); #endif rc = sqlite3_open(zFileName, &db); if( rc!=SQLITE_OK ){ db_err(sqlite3_errmsg(db)); @@ -587,11 +590,11 @@ int rc; const char *zVfs; sqlite3 *db; zVfs = getenv("FOSSIL_VFS"); -#ifdef __MINGW32__ +#if defined(_WIN32) zDbName = sqlite3_win32_mbcs_to_utf8(zDbName); #endif rc = sqlite3_open_v2( zDbName, &db, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, @@ -615,11 +618,11 @@ if( !g.db ){ g.db = openDatabase(zDbName); g.zRepoDb = "main"; db_connection_init(); }else{ -#ifdef __MINGW32__ +#if defined(_WIN32) zDbName = sqlite3_win32_mbcs_to_utf8(zDbName); #endif db_multi_exec("ATTACH DATABASE %Q AS %s", zDbName, zLabel); g.zRepoDb = mprintf("%s", zLabel); } @@ -639,11 +642,11 @@ */ void db_open_config(int useAttach){ char *zDbName; const char *zHome; if( g.configOpen ) return; -#ifdef __MINGW32__ +#if defined(_WIN32) zHome = getenv("LOCALAPPDATA"); if( zHome==0 ){ zHome = getenv("APPDATA"); if( zHome==0 ){ zHome = getenv("HOMEPATH"); @@ -661,17 +664,17 @@ } #endif if( file_isdir(zHome)!=1 ){ fossil_fatal("invalid home directory: %s", zHome); } -#ifndef __MINGW32__ +#ifndef _WIN32 if( access(zHome, W_OK) ){ fossil_fatal("home directory %s must be writeable", zHome); } #endif g.zHome = mprintf("%/", zHome); -#ifdef __MINGW32__ +#if defined(_WIN32) /* . filenames give some window systems problems and many apps problems */ zDbName = mprintf("%//_fossil", zHome); #else zDbName = mprintf("%s/.fossil", zHome); #endif @@ -894,20 +897,30 @@ /* ** Close the database connection. */ void db_close(void){ + sqlite3_stmt *pStmt; if( g.db==0 ) return; while( pAllStmt ){ db_finalize(pAllStmt); } db_end_transaction(1); + pStmt = 0; + while( (pStmt = sqlite3_next_stmt(g.db, pStmt))!=0 ){ + fossil_warning("unfinalized SQL statement: [%s]", sqlite3_sql(pStmt)); + } g.repositoryOpen = 0; g.localOpen = 0; g.configOpen = 0; + sqlite3_wal_checkpoint(g.db, 0); sqlite3_close(g.db); g.db = 0; + if( g.dbConfig ){ + sqlite3_close(g.dbConfig); + g.dbConfig = 0; + } } /* ** Create a new empty repository database with the given name. @@ -934,11 +947,11 @@ zUser = db_get("default-user", 0); if( zUser==0 ){ zUser = zDefaultUser; } if( zUser==0 ){ -#ifdef __MINGW32__ +#if defined(_WIN32) zUser = getenv("USERNAME"); #else zUser = getenv("USER"); #endif } @@ -1000,12 +1013,11 @@ if( zInitialDate ){ int rid; blob_zero(&manifest); blob_appendf(&manifest, "C initial\\sempty\\scheck-in\n"); - zDate = db_text(0, "SELECT datetime(%Q)", zInitialDate); - zDate[10]='T'; + zDate = date_in_standard_format(zInitialDate); blob_appendf(&manifest, "D %s\n", zDate); blob_appendf(&manifest, "P\n"); md5sum_init(); blob_appendf(&manifest, "R %s\n", md5sum_finish(0)); blob_appendf(&manifest, "T *branch * trunk\n"); @@ -1253,11 +1265,10 @@ ** so this routine is a no-op. */ void db_swap_connections(void){ if( !g.useAttach ){ sqlite3 *dbTemp = g.db; - assert( g.dbConfig!=0 ); g.db = g.dbConfig; g.dbConfig = dbTemp; } } @@ -1374,11 +1385,11 @@ db_multi_exec("REPLACE INTO vvar(name,value) VALUES(%Q,%d)", zName, value); } /* ** Record the name of a local repository in the global_config() database. -** The repostiroy filename %s is recorded as an entry with a "name" field +** The repository filename %s is recorded as an entry with a "name" field ** of the following form: ** ** repo:%s ** ** The value field is set to 1. @@ -1481,10 +1492,50 @@ } db_finalize(&q); } +/* +** define all settings, which can be controlled via the set/unset +** command. var is the name of the internal configuration name for db_(un)set. +** If var is 0, the settings name is used. +** width is the length for the edit field on the behavior page, 0 +** is used for on/off checkboxes. +** The behaviour page doesn't use a special layout. It lists all +** set-commands and displays the 'set'-help as info. +*/ +#if INTERFACE +struct stControlSettings { + char const *name; /* Name of the setting */ + char const *var; /* Internal variable name used by db_set() */ + int width; /* Width of display. 0 for boolean values */ + char const *def; /* Default value */ +}; +#endif /* INTERFACE */ +struct stControlSettings const ctrlSettings[] = { + { "auto-captcha", "autocaptcha", 0, "on" }, + { "auto-shun", 0, 0, "on" }, + { "autosync", 0, 0, "on" }, + { "binary-glob", 0, 32, "" }, + { "clearsign", 0, 0, "off" }, + { "diff-command", 0, 16, "" }, + { "dont-push", 0, 0, "off" }, + { "editor", 0, 16, "" }, + { "gdiff-command", 0, 16, "gdiff" }, + { "ignore-glob", 0, 40, "" }, + { "http-port", 0, 16, "8080" }, + { "localauth", 0, 0, "off" }, + { "manifest", 0, 0, "off" }, + { "mtime-changes", 0, 0, "on" }, + { "pgp-command", 0, 32, "gpg --clearsign -o " }, + { "proxy", 0, 32, "off" }, + { "repo-cksum", 0, 0, "on" }, + { "ssh-command", 0, 32, "" }, + { "web-browser", 0, 32, "" }, + { 0,0,0,0 } +}; + /* ** COMMAND: settings ** COMMAND: unset ** %fossil settings ?PROPERTY? ?VALUE? ?-global? ** %fossil unset PROPERTY ?-global? @@ -1499,23 +1550,25 @@ ** auto-captcha If enabled, the Login page provides a button to ** fill in the captcha password. Default: on ** ** auto-shun If enabled, automatically pull the shunning list ** from a server to which the client autosyncs. +** Default: on ** ** autosync If enabled, automatically pull prior to commit ** or update and automatically push after commit or ** tag or branch creation. If the value is "pullonly" ** then only pull operations occur automatically. +** Default: on ** ** binary-glob The VALUE is a comma-separated list of GLOB patterns ** that should be treated as binary files for merging ** purposes. Example: *.xml ** ** clearsign When enabled, fossil will attempt to sign all commits ** with gpg. When disabled (the default), commits will -** be unsigned. +** be unsigned. Default: off ** ** diff-command External command to run when performing a diff. ** If undefined, the internal text diff will be used. ** ** dont-push Prevent this repository from pushing from client to @@ -1536,10 +1589,14 @@ ** localauth If enabled, require that HTTP connections from ** 127.0.0.1 be authenticated by password. If ** false, all HTTP requests from localhost have ** unrestricted access to the repository. ** +** manifest If enabled, automatically create files "manifest" and +** "manifest.uuid" in every checkout. The SQLite and +** Fossil repositories both require this. Default: off. +** ** mtime-changes Use file modification times (mtimes) to detect when ** files have been modified. (Default "on".) ** ** pgp-command Command used to clear-sign manifests at check-in. ** The default is "gpg --clearsign -o ". @@ -1546,35 +1603,25 @@ ** ** proxy URL of the HTTP proxy. If undefined or "off" then ** the "http_proxy" environment variable is consulted. ** If the http_proxy environment variable is undefined ** then a direct HTTP connection is used. +** +** repo-cksum Compute checksums over all files in each checkout +** as a double-check of correctness. Defaults to "on". +** Disable on large repositories for a performance +** improvement. +** +** ssh-command Command used to talk to a remote machine with +** the "ssh://" protocol. ** ** web-browser A shell command used to launch your preferred ** web browser when given a URL as an argument. ** Defaults to "start" on windows, "open" on Mac, ** and "firefox" on Unix. */ void setting_cmd(void){ - static const char *azName[] = { - "auto-captcha", - "auto-shun", - "autosync", - "binary-glob", - "clearsign", - "diff-command", - "dont-push", - "editor", - "gdiff-command", - "ignore-glob", - "http-port", - "localauth", - "mtime-changes", - "pgp-command", - "proxy", - "web-browser", - }; int i; int globalFlag = find_option("global","g",0)!=0; int unsetFlag = g.argv[1][0]=='u'; db_open_config(1); db_find_and_open_repository(0); @@ -1583,28 +1630,80 @@ } if( unsetFlag && g.argc!=3 ){ usage("PROPERTY ?-global?"); } if( g.argc==2 ){ - for(i=0; i=sizeof(azName)/sizeof(azName[0]) ){ + if( !ctrlSettings[i].name ){ fossil_fatal("no such setting: %s", zName); } + isManifest = strcmp(ctrlSettings[i].name, "manifest")==0; + if( isManifest && globalFlag ){ + fossil_fatal("cannot set 'manifest' globally"); + } if( unsetFlag ){ - db_unset(azName[i], globalFlag); + db_unset(ctrlSettings[i].name, globalFlag); }else if( g.argc==4 ){ - db_set(azName[i], g.argv[3], globalFlag); + db_set(ctrlSettings[i].name, g.argv[3], globalFlag); }else{ - print_setting(azName[i]); + isManifest = 0; + print_setting(ctrlSettings[i].name); + } + if( isManifest ){ + manifest_to_disk(db_lget_int("checkout", 0)); } }else{ usage("?PROPERTY? ?VALUE?"); } } + +/* +** The input in a a timespan measured in days. Return a string which +** describes that timespan in units of seconds, minutes, hours, days, +** or years, depending on its duration. +*/ +char *db_timespan_name(double rSpan){ + if( rSpan<0 ) rSpan = -rSpan; + rSpan *= 24.0*3600.0; /* Convert units to seconds */ + if( rSpan<120.0 ){ + return sqlite3_mprintf("%.1f seconds", rSpan); + } + rSpan /= 60.0; /* Convert units to minutes */ + if( rSpan<90.0 ){ + return sqlite3_mprintf("%.1f minutes", rSpan); + } + rSpan /= 60.0; /* Convert units to hours */ + if( rSpan<=48.0 ){ + return sqlite3_mprintf("%.1f hours", rSpan); + } + rSpan /= 24.0; /* Convert units to days */ + if( rSpan<=365.0 ){ + return sqlite3_mprintf("%.1f days", rSpan); + } + rSpan /= 356.24; /* Convert units to years */ + return sqlite3_mprintf("%.1f years", rSpan); +} + +/* +** COMMAND: test-timespan +** %fossil test-timespan TIMESTAMP +** +** Print the approximate span of time from now to TIMESTAMP. +*/ +void test_timespan_cmd(void){ + double rDiff; + if( g.argc!=3 ) usage("TIMESTAMP"); + sqlite3_open(":memory:", &g.db); + rDiff = db_double(0.0, "SELECT julianday('now') - julianday(%Q)", g.argv[2]); + printf("Time differences: %s\n", db_timespan_name(rDiff)); + sqlite3_close(g.db); + g.db = 0; +} Index: src/delta.c ================================================================== --- src/delta.c +++ src/delta.c @@ -195,31 +195,38 @@ /* ** Compute a 32-bit checksum on the N-byte buffer. Return the result. */ static unsigned int checksum(const char *zIn, size_t N){ const unsigned char *z = (const unsigned char *)zIn; - unsigned sum = 0; + unsigned sum0 = 0; + unsigned sum1 = 0; + unsigned sum2 = 0; + unsigned sum3 = 0; while(N >= 16){ - sum += ((unsigned)z[0] + z[4] + z[8] + z[12]) << 24; - sum += ((unsigned)z[1] + z[5] + z[9] + z[13]) << 16; - sum += ((unsigned)z[2] + z[6] + z[10]+ z[14]) << 8; - sum += ((unsigned)z[3] + z[7] + z[11]+ z[15]); + sum0 += ((unsigned)z[0] + z[4] + z[8] + z[12]); + sum1 += ((unsigned)z[1] + z[5] + z[9] + z[13]); + sum2 += ((unsigned)z[2] + z[6] + z[10]+ z[14]); + sum3 += ((unsigned)z[3] + z[7] + z[11]+ z[15]); z += 16; N -= 16; } while(N >= 4){ - sum += (z[0]<<24) | (z[1]<<16) | (z[2]<<8) | z[3]; + sum0 += z[0]; + sum1 += z[1]; + sum2 += z[2]; + sum3 += z[3]; z += 4; N -= 4; } + sum3 += (sum2 << 8) + (sum1 << 16) + (sum0 << 24); switch(N){ - case 3: sum += (z[2] << 8); - case 2: sum += (z[1] << 16); - case 1: sum += (z[0] << 24); + case 3: sum3 += (z[2] << 8); + case 2: sum3 += (z[1] << 16); + case 1: sum3 += (z[0] << 24); default: ; } - return sum; + return sum3; } /* ** Create a new delta. ** @@ -317,12 +324,11 @@ /* Compute the hash table used to locate matching sections in the ** source file. */ nHash = lenSrc/NHASH; - collide = malloc( nHash*2*sizeof(int) ); - if( collide==0 ) return -1; + collide = fossil_malloc( nHash*2*sizeof(int) ); landmark = &collide[nHash]; memset(landmark, -1, nHash*sizeof(int)); memset(collide, -1, nHash*sizeof(int)); for(i=0; i - @

A leaf is a check-in with no descendants.

- @

An open leaf is a leaf that does not have a "closed" tag + @

leaf

+ @ is a check-in with no descendants.

+ @

open leaf

+ @ is a leaf that does not have a "closed" tag @ and is thus assumed to still be in use.

- @

A closed leaf has a "closed" tag and is thus assumed to + @

closed leaf

+ @ has a "closed" tag and is thus assumed to @ be historical and no longer in active use.

@ style_sidebox_end(); if( showAll ){ @@ -341,14 +344,14 @@ " ORDER BY event.mtime DESC", timeline_query_for_www() ); www_print_timeline(&q, TIMELINE_LEAFONLY, leaves_extra); db_finalize(&q); - @
- @ style_footer(); } Index: src/diff.c ================================================================== --- src/diff.c +++ src/diff.c @@ -65,16 +65,18 @@ /* ** Return an array of DLine objects containing a pointer to the ** start of each line and a hash of that line. The lower ** bits of the hash store the length of each line. ** -** Trailing whitespace is removed from each line. +** Trailing whitespace is removed from each line. 2010-08-20: Not any +** more. If trailing whitespace is ignored, the "patch" command gets +** confused by the diff output. Ticket [a9f7b23c2e376af5b0e5b] ** ** Return 0 if the file is binary or contains a line that is ** too long. */ -static DLine *break_into_lines(const char *z, int n, int *pnLine){ +static DLine *break_into_lines(const char *z, int n, int *pnLine, int ignoreWS){ int nLine, i, j, k, x; unsigned int h, h2; DLine *a; /* Count the number of lines. Allocate space to hold @@ -94,19 +96,23 @@ } } if( j>LENGTH_MASK ){ return 0; } - a = malloc( nLine*sizeof(a[0]) ); - if( a==0 ) fossil_panic("out of memory"); + a = fossil_malloc( nLine*sizeof(a[0]) ); memset(a, 0, nLine*sizeof(a[0]) ); + if( n==0 ){ + *pnLine = 0; + return a; + } /* Fill in the array */ for(i=0; i0 && isspace(z[k-1]); k--){} + k = j; + while( ignoreWS && k>0 && fossil_isspace(z[k-1]) ){ k--; } for(h=0, x=0; xaEdit, nEdit*sizeof(int)); - if( a==0 ){ - free( p->aEdit ); - p->nEdit = 0; - nEdit = 0; - } - p->aEdit = a; + p->aEdit = fossil_realloc(p->aEdit, nEdit*sizeof(int)); p->nEditAlloc = nEdit; } /* ** Append a new COPY/DELETE/INSERT triple. @@ -279,16 +278,70 @@ for(j=0; jaFrom[] */ + int iS2, int iE2, /* Range of lines in p->aTo[] */ + int *piSX, int *piEX, /* Write p->aFrom[] common segment here */ + int *piSY, int *piEY /* Write p->aTo[] common segment here */ +){ + int mxLength = 0; /* Length of longest common subsequence */ + int i, j; /* Loop counters */ + int k; /* Length of a candidate subsequence */ + int iSXb = iS1; /* Best match so far */ + int iSYb = iS2; /* Best match so far */ + + for(i=iS1; iaFrom[i], &p->aTo[j]) ) continue; + if( mxLength && !same_dline(&p->aFrom[i+mxLength], &p->aTo[j+mxLength]) ){ + continue; + } + k = 1; + while( i+kaFrom[i+k],&p->aTo[j+k]) ){ + k++; + } + if( k>mxLength ){ + iSXb = i; + iSYb = j; + mxLength = k; + } + } + } + *piSX = iSXb; + *piEX = iSXb + mxLength; + *piSY = iSYb; + *piEY = iSYb + mxLength; +} /* ** Compare two blocks of text on lines iS1 through iE1-1 of the aFrom[] ** file and lines iS2 through iE2-1 of the aTo[] file. Locate a sequence ** of lines in these two blocks that are exactly the same. Return ** the bounds of the matching sequence. +** +** If there are two or more possible answers of the same length, the +** returned sequence should be the one closest to the center of the +** input range. +** +** Ideally, the common sequence should be the longest possible common +** sequence. However, an exact computation of LCS is O(N*N) which is +** way too slow for larger files. So this routine uses an O(N) +** heuristic approximation based on hashing that usually works about +** as well. But if the O(N) algorithm doesn't get a good solution +** and N is not too large, we fall back to an exact solution by +** calling optimalLCS(). */ static void longestCommonSequence( DContext *p, /* Two files being compared */ int iS1, int iE1, /* Range of lines in p->aFrom[] */ int iS2, int iE2, /* Range of lines in p->aTo[] */ @@ -302,10 +355,11 @@ int skew; /* How lopsided is the match */ int dist; /* Distance of match from center */ int mid; /* Center of the span */ int iSXb, iSYb, iEXb, iEYb; /* Best match so far */ int iSXp, iSYp, iEXp, iEYp; /* Previous match */ + iSXb = iSXp = iS1; iEXb = iEXp = iS1; iSYb = iSYp = iS2; iEYb = iEYp = iS2; @@ -354,14 +408,20 @@ iSYp = iSY; iEXp = iEX; iEYp = iEY; } } - *piSX = iSXb; - *piSY = iSYb; - *piEX = iEXb; - *piEY = iEYb; + if( iSXb==iEXb && (iE1-iS1)*(iE2-iS2)<400 ){ + /* If no common sequence is found using the hashing heuristic and + ** the input is not too big, use the expensive exact solution */ + optimalLCS(p, iS1, iE1, iS2, iE2, piSX, piEX, piSY, piEY); + }else{ + *piSX = iSXb; + *piSY = iSYb; + *piEX = iEXb; + *piEY = iEYb; + } /* printf("LCS(%d..%d/%d..%d) = %d..%d/%d..%d\n", iS1, iE1, iS2, iE2, *piSX, *piEX, *piSY, *piEY); */ } /* @@ -472,18 +532,21 @@ */ int *text_diff( Blob *pA_Blob, /* FROM file */ Blob *pB_Blob, /* TO file */ Blob *pOut, /* Write unified diff here if not NULL */ - int nContext /* Amount of context to unified diff */ + int nContext, /* Amount of context to unified diff */ + int ignoreEolWs /* Ignore whitespace at the end of lines */ ){ DContext c; /* Prepare the input files */ memset(&c, 0, sizeof(c)); - c.aFrom = break_into_lines(blob_str(pA_Blob), blob_size(pA_Blob), &c.nFrom); - c.aTo = break_into_lines(blob_str(pB_Blob), blob_size(pB_Blob), &c.nTo); + c.aFrom = break_into_lines(blob_str(pA_Blob), blob_size(pA_Blob), + &c.nFrom, ignoreEolWs); + c.aTo = break_into_lines(blob_str(pB_Blob), blob_size(pB_Blob), + &c.nTo, ignoreEolWs); if( c.aFrom==0 || c.aTo==0 ){ free(c.aFrom); free(c.aTo); if( pOut ){ blob_appendf(pOut, "cannot compute difference between binary files\n"); @@ -522,11 +585,11 @@ if( g.argc<4 ) usage("FILE1 FILE2 ..."); blob_read_from_file(&a, g.argv[2]); for(i=3; i3 ) printf("-------------------------------\n"); blob_read_from_file(&b, g.argv[i]); - R = text_diff(&a, &b, 0, 0); + R = text_diff(&a, &b, 0, 0, 0); for(r=0; R[r] || R[r+1] || R[r+2]; r += 3){ printf(" copy %4d delete %4d insert %4d\n", R[r], R[r+1], R[r+2]); } /* free(R); */ blob_reset(&b); @@ -540,11 +603,11 @@ Blob a, b, out; if( g.argc!=4 ) usage("FILE1 FILE2"); blob_read_from_file(&a, g.argv[2]); blob_read_from_file(&b, g.argv[3]); blob_zero(&out); - text_diff(&a, &b, &out, 3); + text_diff(&a, &b, &out, 3, 0); blob_write_to_file(&out, "-"); } /************************************************************************** ** The basic difference engine is above. What follows is the annotation @@ -574,16 +637,15 @@ */ static int annotation_start(Annotator *p, Blob *pInput){ int i; memset(p, 0, sizeof(*p)); - p->c.aTo = break_into_lines(blob_str(pInput), blob_size(pInput), &p->c.nTo); + p->c.aTo = break_into_lines(blob_str(pInput), blob_size(pInput),&p->c.nTo,1); if( p->c.aTo==0 ){ return 1; } - p->aOrig = malloc( sizeof(p->aOrig[0])*p->c.nTo ); - if( p->aOrig==0 ) fossil_panic("out of memory"); + p->aOrig = fossil_malloc( sizeof(p->aOrig[0])*p->c.nTo ); for(i=0; ic.nTo; i++){ p->aOrig[i].z = p->c.aTo[i].z; p->aOrig[i].n = p->c.aTo[i].h & LENGTH_MASK; p->aOrig[i].zSrc = 0; } @@ -602,11 +664,11 @@ int i, j; int lnTo; /* Prepare the parent file to be diffed */ p->c.aFrom = break_into_lines(blob_str(pParent), blob_size(pParent), - &p->c.nFrom); + &p->c.nFrom, 1); if( p->c.aFrom==0 ){ return 1; } /* Compute the differences going from pParent to the file being @@ -757,11 +819,11 @@ ** COMMAND: annotate ** ** %fossil annotate FILENAME ** ** Output the text of a file with markings to show when each line of -** the file was introduced. +** the file was last modified. */ void annotate_cmd(void){ int fnid; /* Filename ID */ int fid; /* File instance ID */ int mid; /* Manifest where file was checked in */ Index: src/diffcmd.c ================================================================== --- src/diffcmd.c +++ src/diffcmd.c @@ -20,49 +20,14 @@ #include "config.h" #include "diffcmd.h" #include /* -** Shell-escape the given string. Append the result to a blob. -*/ -static void shell_escape(Blob *pBlob, const char *zIn){ - int n = blob_size(pBlob); - int k = strlen(zIn); - int i, c; - char *z; - for(i=0; (c = zIn[i])!=0; i++){ - if( isspace(c) || c=='"' || (c=='\\' && zIn[i+1]!=0) ){ - blob_appendf(pBlob, "\"%s\"", zIn); - z = blob_buffer(pBlob); - for(i=n+1; i<=n+k; i++){ - if( z[i]=='"' ) z[i] = '_'; - } - return; - } - } - blob_append(pBlob, zIn, -1); -} - -/* -** This function implements a cross-platform "system()" interface. -*/ -int portable_system(const char *zOrigCmd){ - int rc; -#ifdef __MINGW32__ - /* On windows, we have to put double-quotes around the entire command. - ** Who knows why - this is just the way windows works. - */ - char *zNewCmd = mprintf("\"%s\"", zOrigCmd); - rc = system(zNewCmd); - free(zNewCmd); -#else - /* On unix, evaluate the command directly. - */ - rc = system(zOrigCmd); -#endif - return rc; -} +** Diff option flags +*/ +#define DIFF_NEWFILE 0x01 /* Treat non-existing fails as empty files */ +#define DIFF_NOEOLWS 0x02 /* Ignore whitespace at the end of lines */ /* ** Show the difference between two files, one in memory and one on disk. ** ** The difference is the set of edits needed to transform pFile1 into @@ -73,24 +38,31 @@ */ static void diff_file( Blob *pFile1, /* In memory content to compare from */ const char *zFile2, /* On disk content to compare to */ const char *zName, /* Display name of the file */ - const char *zDiffCmd /* Command for comparison */ + const char *zDiffCmd, /* Command for comparison */ + int ignoreEolWs /* Ignore whitespace at end of line */ ){ if( zDiffCmd==0 ){ - Blob out; /* Diff output text */ - Blob file2; /* Content of zFile2 */ + Blob out; /* Diff output text */ + Blob file2; /* Content of zFile2 */ + const char *zName2; /* Name of zFile2 for display */ /* Read content of zFile2 into memory */ blob_zero(&file2); - blob_read_from_file(&file2, zFile2); + if( file_size(zFile2)<0 ){ + zName2 = "/dev/null"; + }else{ + blob_read_from_file(&file2, zFile2); + zName2 = zName; + } /* Compute and output the differences */ blob_zero(&out); - text_diff(pFile1, &file2, &out, 5); - printf("--- %s\n+++ %s\n", zName, zName); + text_diff(pFile1, &file2, &out, 5, ignoreEolWs); + printf("--- %s\n+++ %s\n", zName, zName2); printf("%s\n", blob_str(&out)); /* Release memory resources */ blob_reset(&file2); blob_reset(&out); @@ -114,11 +86,11 @@ shell_escape(&cmd, blob_str(&nameFile1)); blob_append(&cmd, " ", 1); shell_escape(&cmd, zFile2); /* Run the external diff command */ - portable_system(blob_str(&cmd)); + fossil_system(blob_str(&cmd)); /* Delete the temporary file and clean up memory used */ unlink(blob_str(&nameFile1)); blob_reset(&nameFile1); blob_reset(&cmd); @@ -136,17 +108,18 @@ */ static void diff_file_mem( Blob *pFile1, /* In memory content to compare from */ Blob *pFile2, /* In memory content to compare to */ const char *zName, /* Display name of the file */ - const char *zDiffCmd /* Command for comparison */ + const char *zDiffCmd, /* Command for comparison */ + int ignoreEolWs /* Ignore whitespace at end of lines */ ){ if( zDiffCmd==0 ){ Blob out; /* Diff output text */ blob_zero(&out); - text_diff(pFile1, pFile2, &out, 5); + text_diff(pFile1, pFile2, &out, 5, ignoreEolWs); printf("--- %s\n+++ %s\n", zName, zName); printf("%s\n", blob_str(&out)); /* Release memory resources */ blob_reset(&out); @@ -167,11 +140,11 @@ shell_escape(&cmd, zTemp1); blob_append(&cmd, " ", 1); shell_escape(&cmd, zTemp2); /* Run the external diff command */ - portable_system(blob_str(&cmd)); + fossil_system(blob_str(&cmd)); /* Delete the temporary file and clean up memory used */ unlink(zTemp1); unlink(zTemp2); blob_reset(&cmd); @@ -180,30 +153,42 @@ /* ** Do a diff against a single file named in g.argv[2] from version zFrom ** against the same file on disk. */ -static void diff_one_against_disk(const char *zFrom, const char *zDiffCmd){ +static void diff_one_against_disk( + const char *zFrom, /* Name of file */ + const char *zDiffCmd, /* Use this "diff" command */ + int ignoreEolWs /* Ignore whitespace changes at end of lines */ +){ Blob fname; Blob content; file_tree_name(g.argv[2], &fname, 1); historical_version_of_file(zFrom, blob_str(&fname), &content, 0); - diff_file(&content, g.argv[2], g.argv[2], zDiffCmd); + diff_file(&content, g.argv[2], g.argv[2], zDiffCmd, ignoreEolWs); blob_reset(&content); blob_reset(&fname); } /* ** Run a diff between the version zFrom and files on disk. zFrom might ** be NULL which means to simply show the difference between the edited ** files on disk and the check-out on which they are based. */ -static void diff_all_against_disk(const char *zFrom, const char *zDiffCmd){ +static void diff_all_against_disk( + const char *zFrom, /* Version to difference from */ + const char *zDiffCmd, /* Use this diff command. NULL for built-in */ + int diffFlags /* Flags controlling diff output */ +){ int vid; Blob sql; Stmt q; + int ignoreEolWs; /* Ignore end-of-line whitespace */ + int asNewFile; /* Treat non-existant files as empty files */ + ignoreEolWs = (diffFlags & DIFF_NOEOLWS)!=0; + asNewFile = (diffFlags & DIFF_NEWFILE)!=0; vid = db_lget_int("checkout", 0); vfile_check_signature(vid, 1); blob_zero(&sql); db_begin_transaction(); if( zFrom ){ @@ -246,33 +231,44 @@ while( db_step(&q)==SQLITE_ROW ){ const char *zPathname = db_column_text(&q,0); int isDeleted = db_column_int(&q, 1); int isChnged = db_column_int(&q,2); int isNew = db_column_int(&q,3); + int srcid = db_column_int(&q, 4); char *zFullName = mprintf("%s%s", g.zLocalRoot, zPathname); + char *zToFree = zFullName; + int showDiff = 1; if( isDeleted ){ printf("DELETED %s\n", zPathname); + if( !asNewFile ){ showDiff = 0; zFullName = "/dev/null"; } }else if( access(zFullName, 0) ){ printf("MISSING %s\n", zPathname); + if( !asNewFile ){ showDiff = 0; } }else if( isNew ){ printf("ADDED %s\n", zPathname); - }else if( isDeleted ){ - printf("DELETED %s\n", zPathname); + srcid = 0; + if( !asNewFile ){ showDiff = 0; } }else if( isChnged==3 ){ printf("ADDED_BY_MERGE %s\n", zPathname); - }else{ - int srcid = db_column_int(&q, 4); + srcid = 0; + if( !asNewFile ){ showDiff = 0; } + } + if( showDiff ){ Blob content; - content_get(srcid, &content); + if( srcid>0 ){ + content_get(srcid, &content); + }else{ + blob_zero(&content); + } printf("Index: %s\n=======================================" "============================\n", zPathname ); - diff_file(&content, zFullName, zPathname, zDiffCmd); + diff_file(&content, zFullName, zPathname, zDiffCmd, ignoreEolWs); blob_reset(&content); } - free(zFullName); + free(zToFree); } db_finalize(&q); db_end_transaction(1); /* ROLLBACK */ } @@ -282,75 +278,113 @@ ** The filename is contained in g.argv[2]. */ static void diff_one_two_versions( const char *zFrom, const char *zTo, - const char *zDiffCmd + const char *zDiffCmd, + int ignoreEolWs ){ char *zName; Blob fname; Blob v1, v2; file_tree_name(g.argv[2], &fname, 1); zName = blob_str(&fname); historical_version_of_file(zFrom, zName, &v1, 0); historical_version_of_file(zTo, zName, &v2, 0); - diff_file_mem(&v1, &v2, zName, zDiffCmd); + diff_file_mem(&v1, &v2, zName, zDiffCmd, ignoreEolWs); blob_reset(&v1); blob_reset(&v2); blob_reset(&fname); } + +/* +** Show the difference between two files identified by ManifestFile +** entries. +*/ +static void diff_manifest_entry( + struct ManifestFile *pFrom, + struct ManifestFile *pTo, + const char *zDiffCmd, + int ignoreEolWs +){ + Blob f1, f2; + int rid; + const char *zName = pFrom ? pFrom->zName : pTo->zName; + printf("Index: %s\n=======================================" + "============================\n", zName); + if( pFrom ){ + rid = uuid_to_rid(pFrom->zUuid, 0); + content_get(rid, &f1); + }else{ + blob_zero(&f1); + } + if( pTo ){ + rid = uuid_to_rid(pTo->zUuid, 0); + content_get(rid, &f2); + }else{ + blob_zero(&f2); + } + diff_file_mem(&f1, &f2, zName, zDiffCmd, ignoreEolWs); + blob_reset(&f1); + blob_reset(&f2); +} /* ** Output the differences between two check-ins. */ static void diff_all_two_versions( const char *zFrom, const char *zTo, - const char *zDiffCmd + const char *zDiffCmd, + int diffFlags ){ - Manifest mFrom, mTo; - int iFrom, iTo; + Manifest *pFrom, *pTo; + ManifestFile *pFromFile, *pToFile; + int ignoreEolWs = (diffFlags & DIFF_NOEOLWS)!=0 ? 1 : 0; + int asNewFlag = (diffFlags & DIFF_NEWFILE)!=0 ? 1 : 0; + + pFrom = manifest_get_by_name(zFrom, 0); + manifest_file_rewind(pFrom); + pFromFile = manifest_file_next(pFrom,0); + pTo = manifest_get_by_name(zTo, 0); + manifest_file_rewind(pTo); + pToFile = manifest_file_next(pTo,0); - manifest_from_name(zFrom, &mFrom); - manifest_from_name(zTo, &mTo); - iFrom = iTo = 0; - while( iFrom=mFrom.nFile ){ + if( pFromFile==0 ){ cmp = +1; - }else if( iTo>=mTo.nFile ){ + }else if( pToFile==0 ){ cmp = -1; }else{ - cmp = strcmp(mFrom.aFile[iFrom].zName, mTo.aFile[iTo].zName); + cmp = strcmp(pFromFile->zName, pToFile->zName); } if( cmp<0 ){ - printf("DELETED %s\n", mFrom.aFile[iFrom].zName); - iFrom++; - }else if( cmp>0 ){ - printf("ADDED %s\n", mTo.aFile[iTo].zName); - iTo++; - }else if( strcmp(mFrom.aFile[iFrom].zUuid, mTo.aFile[iTo].zUuid)==0 ){ - /* No changes */ - iFrom++; - iTo++; - }else{ - Blob f1, f2; - int rid; - printf("CHANGED %s\n", mFrom.aFile[iFrom].zName); - rid = uuid_to_rid(mFrom.aFile[iFrom].zUuid, 0); - content_get(rid, &f1); - rid = uuid_to_rid(mTo.aFile[iTo].zUuid, 0); - content_get(rid, &f2); - diff_file_mem(&f1, &f2, mFrom.aFile[iFrom].zName, zDiffCmd); - blob_reset(&f1); - blob_reset(&f2); - iFrom++; - iTo++; - } - } - manifest_clear(&mFrom); - manifest_clear(&mTo); + printf("DELETED %s\n", pFromFile->zName); + if( asNewFlag ){ + diff_manifest_entry(pFromFile, 0, zDiffCmd, ignoreEolWs); + } + pFromFile = manifest_file_next(pFrom,0); + }else if( cmp>0 ){ + printf("ADDED %s\n", pToFile->zName); + if( asNewFlag ){ + diff_manifest_entry(0, pToFile, zDiffCmd, ignoreEolWs); + } + pToFile = manifest_file_next(pTo,0); + }else if( strcmp(pFromFile->zUuid, pToFile->zUuid)==0 ){ + /* No changes */ + pFromFile = manifest_file_next(pFrom,0); + pToFile = manifest_file_next(pTo,0); + }else{ + printf("CHANGED %s\n", pFromFile->zName); + diff_manifest_entry(pFromFile, pToFile, zDiffCmd, ignoreEolWs); + pFromFile = manifest_file_next(pFrom,0); + pToFile = manifest_file_next(pTo,0); + } + } + manifest_destroy(pFrom); + manifest_destroy(pTo); } /* ** COMMAND: diff ** COMMAND: gdiff @@ -373,44 +407,52 @@ ** ** The "-i" command-line option forces the use of the internal diff logic ** rather than any external diff program that might be configured using ** the "setting" command. If no external diff program is configured, then ** the "-i" option is a no-op. The "-i" option converts "gdiff" into "diff". +** +** The "-N" or "--new-file" option causes the complete text of added or +** deleted files to be displayed. */ void diff_cmd(void){ int isGDiff; /* True for gdiff. False for normal diff */ int isInternDiff; /* True for internal diff */ + int hasNFlag; /* True if -N or --new-file flag is used */ const char *zFrom; /* Source version number */ const char *zTo; /* Target version number */ const char *zDiffCmd = 0; /* External diff command. NULL for internal diff */ + int diffFlags = 0; /* Flags to control the DIFF */ isGDiff = g.argv[1][0]=='g'; isInternDiff = find_option("internal","i",0)!=0; zFrom = find_option("from", "r", 1); zTo = find_option("to", 0, 1); + hasNFlag = find_option("new-file","N",0)!=0; + + if( hasNFlag ) diffFlags |= DIFF_NEWFILE; if( zTo==0 ){ db_must_be_within_tree(); verify_all_options(); - if( !isInternDiff && g.argc==3 ){ + if( !isInternDiff ){ zDiffCmd = db_get(isGDiff ? "gdiff-command" : "diff-command", 0); } if( g.argc==3 ){ - diff_one_against_disk(zFrom, zDiffCmd); + diff_one_against_disk(zFrom, zDiffCmd, 0); }else{ - diff_all_against_disk(zFrom, zDiffCmd); + diff_all_against_disk(zFrom, zDiffCmd, diffFlags); } }else if( zFrom==0 ){ fossil_fatal("must use --from if --to is present"); }else{ db_find_and_open_repository(1); verify_all_options(); - if( !isInternDiff && g.argc==3 ){ + if( !isInternDiff ){ zDiffCmd = db_get(isGDiff ? "gdiff-command" : "diff-command", 0); } if( g.argc==3 ){ - diff_one_two_versions(zFrom, zTo, zDiffCmd); + diff_one_two_versions(zFrom, zTo, zDiffCmd, 0); }else{ - diff_all_two_versions(zFrom, zTo, zDiffCmd); + diff_all_two_versions(zFrom, zTo, zDiffCmd, diffFlags); } } } Index: src/doc.c ================================================================== --- src/doc.c +++ src/doc.c @@ -290,11 +290,11 @@ if( zName[i]=='.' ) z = &zName[i+1]; } len = strlen(z); if( len10000 ){ db_multi_exec("DELETE FROM vcache"); } - if( content_get(vid, &baseline)==0 ){ - goto doc_not_found; - } - if( manifest_parse(&m, &baseline)==0 || m.type!=CFTYPE_MANIFEST ){ + pM = manifest_get(vid, CFTYPE_MANIFEST); + if( pM==0 ){ goto doc_not_found; } db_prepare(&s, "INSERT INTO vcache(vid,fname,rid)" " SELECT %d, :fname, rid FROM blob" " WHERE uuid=:uuid", vid ); - for(i=0; izName); + db_bind_text(&s, ":uuid", pFile->zUuid); db_step(&s); db_reset(&s); } db_finalize(&s); - manifest_clear(&m); + manifest_destroy(pM); /* Try again to find the file */ rid = db_int(0, "SELECT rid FROM vcache" " WHERE vid=%d AND fname=%Q", vid, zName); } Index: src/encode.c ================================================================== --- src/encode.c +++ src/encode.c @@ -44,12 +44,11 @@ default: count++; break; } i++; } i = 0; - zOut = malloc( count+1 ); - if( zOut==0 ) return 0; + zOut = fossil_malloc( count+1 ); while( n-->0 && (c = *zIn)!=0 ){ switch( c ){ case '<': zOut[i++] = '&'; zOut[i++] = 'l'; @@ -100,11 +99,12 @@ int i = 0; int count = 0; char *zOut; int other; # define IsSafeChar(X) \ - (isalnum(X) || (X)=='.' || (X)=='$' || (X)=='-' || (X)=='_' || (X)==other) + (fossil_isalnum(X) || (X)=='.' || (X)=='$' \ + || (X)=='~' || (X)=='-' || (X)=='_' || (X)==other) if( zIn==0 ) return 0; if( n<0 ) n = strlen(zIn); other = encodeSlash ? 'a' : '/'; while( i0 && (c = *zIn)!=0 ){ if( IsSafeChar(c) ){ zOut[i++] = c; }else if( c==' ' ){ zOut[i++] = '+'; @@ -233,21 +232,21 @@ c = zIn[i]; if( c==0 || c==' ' || c=='\n' || c=='\t' || c=='\r' || c=='\f' || c=='\v' || c=='\\' ) n++; } n += nIn; - zOut = malloc( n+1 ); + zOut = fossil_malloc( n+1 ); if( zOut ){ for(i=j=0; i>2) & 0x3f ]; z64[n++] = zBase[ ((zData[i]<<4) & 0x30) | ((zData[i+1]>>4) & 0x0f) ]; z64[n++] = zBase[ ((zData[i+1]<<2) & 0x3c) | ((zData[i+2]>>6) & 0x03) ]; z64[n++] = zBase[ zData[i+2] & 0x3f ]; @@ -370,11 +370,11 @@ for(i=0; zBase[i]; i++){ trans[zBase[i] & 0x7f] = i; } isInit = 1; } n64 = strlen(z64); while( n64>0 && z64[n64-1]=='=' ) n64--; - zData = malloc( (n64*3)/4 + 4 ); + zData = fossil_malloc( (n64*3)/4 + 4 ); for(i=j=0; i+3 %s (%s)\n", g.argv[i], z, z2); + free(z); + free(z2); + z = unobscure(g.argv[i]); + printf("UNOBSCURE: %s -> %s\n", g.argv[i], z); + free(z); + } +} ADDED src/event.c Index: src/event.c ================================================================== --- src/event.c +++ src/event.c @@ -0,0 +1,438 @@ +/* +** Copyright (c) 2010 D. Richard Hipp +** +** This program is free software; you can redistribute it and/or +** modify it under the terms of the Simplified BSD License (also +** known as the "2-Clause License" or "FreeBSD License".) + +** This program is distributed in the hope that it will be useful, +** but without any warranty; without even the implied warranty of +** merchantability or fitness for a particular purpose. +** +** Author contact information: +** drh@hwaci.com +** http://www.hwaci.com/drh/ +** +******************************************************************************* +** +** This file contains code to do formatting of event messages: +** +** Milestones +** Blog posts +** New articles +** Process checkpoints +** Announcements +*/ +#include +#include +#include "config.h" +#include "event.h" + +/* +** Output a hyperlink to an event given its tagid. +*/ +void hyperlink_to_event_tagid(int tagid){ + char *zEventId; + char zShort[12]; + + zEventId = db_text(0, "SELECT substr(tagname, 7) FROM tag WHERE tagid=%d", + tagid); + sqlite3_snprintf(sizeof(zShort), zShort, "%.10s", zEventId); + if( g.okHistory ){ + @ [%s(zShort)] + }else{ + @ [%s(zShort)] + } + free(zEventId); +} + +/* +** WEBPAGE: event +** URL: /event +** PARAMETERS: +** +** name=EVENTID // Identify the event to display EVENTID must be complete +** detail=BOOLEAN // Show details if TRUE. Default is FALSE. Optional. +** aid=ARTIFACTID // Which specific version of the event. Optional. +** +** Display an existing event identified by EVENTID +*/ +void event_page(void){ + int rid = 0; /* rid of the event artifact */ + char *zUuid; /* UUID corresponding to rid */ + const char *zEventId; /* Event identifier */ + char *zETime; /* Time of the event */ + char *zATime; /* Time the artifact was created */ + int specRid; /* rid specified by aid= parameter */ + int prevRid, nextRid; /* Previous or next edits of this event */ + Manifest *pEvent; /* Parsed event artifact */ + Blob fullbody; /* Complete content of the event body */ + Blob title; /* Title extracted from the event body */ + Blob tail; /* Event body that comes after the title */ + Stmt q1; /* Query to search for the event */ + int showDetail; /* True to show details */ + + + /* wiki-read privilege is needed in order to read events. + */ + login_check_credentials(); + if( !g.okRdWiki ){ + login_needed(); + return; + } + + zEventId = P("name"); + if( zEventId==0 ){ fossil_redirect_home(); return; } + zUuid = (char*)P("aid"); + specRid = zUuid ? uuid_to_rid(zUuid, 0) : 0; + rid = nextRid = prevRid = 0; + db_prepare(&q1, + "SELECT rid FROM tagxref" + " WHERE tagid=(SELECT tagid FROM tag WHERE tagname GLOB 'event-%q*')" + " ORDER BY mtime DESC", + zEventId + ); + while( db_step(&q1)==SQLITE_ROW ){ + nextRid = rid; + rid = db_column_int(&q1, 0); + if( specRid==0 || specRid==rid ){ + if( db_step(&q1)==SQLITE_ROW ){ + prevRid = db_column_int(&q1, 0); + } + break; + } + } + db_finalize(&q1); + if( rid==0 || (specRid!=0 && specRid!=rid) ){ + style_header("No Such Event"); + @ Cannot locate specified event + style_footer(); + return; + } + zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rid); + showDetail = atoi(PD("detail","0")); + + /* Extract the event content. + */ + pEvent = manifest_get(rid, CFTYPE_EVENT); + if( pEvent==0 ){ + fossil_panic("Object #%d is not an event", rid); + } + blob_init(&fullbody, pEvent->zWiki, -1); + if( wiki_find_title(&fullbody, &title, &tail) ){ + style_header(blob_str(&title)); + }else{ + style_header("Event %S", zEventId); + tail = fullbody; + } + if( g.okWrWiki && g.okWrite && nextRid==0 ){ + style_submenu_element("Edit", "Edit", "%s/eventedit?name=%s", + g.zTop, zEventId); + } + zETime = db_text(0, "SELECT datetime(%.17g)", pEvent->rEventDate); + style_submenu_element("Context", "Context", "%s/timeline?c=%T", + g.zTop, zETime); + if( g.okHistory ){ + if( showDetail ){ + style_submenu_element("Plain", "Plain", "%s/event?name=%s&aid=%s", + g.zTop, zEventId, zUuid); + if( nextRid ){ + char *zNext; + zNext = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", nextRid); + style_submenu_element("Next", "Next", + "%s/event?name=%s&aid=%s&detail=1", + g.zTop, zEventId, zNext); + free(zNext); + } + if( prevRid ){ + char *zPrev; + zPrev = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", prevRid); + style_submenu_element("Prev", "Prev", + "%s/event?name=%s&aid=%s&detail=1", + g.zTop, zEventId, zPrev); + free(zPrev); + } + }else{ + style_submenu_element("Detail", "Detail", + "%s/event?name=%s&aid=%s&detail=1", + g.zTop, zEventId, zUuid); + } + } + + if( showDetail && g.okHistory ){ + int i; + const char *zClr = 0; + Blob comment; + + zATime = db_text(0, "SELECT datetime(%.17g)", pEvent->rDate); + @

Event [%S(zUuid)] at + @ [%s(zETime)] + @ entered by user %h(pEvent->zUser) on + @ [%s(zATime)]:

+ @

+ for(i=0; inTag; i++){ + if( strcmp(pEvent->aTag[i].zName,"+bgcolor")==0 ){ + zClr = pEvent->aTag[i].zValue; + } + } + if( zClr && zClr[0]==0 ) zClr = 0; + if( zClr ){ + @
+ }else{ + @
+ } + blob_init(&comment, pEvent->zComment, -1); + wiki_convert(&comment, 0, WIKI_INLINE); + blob_reset(&comment); + @
+ @

+ } + + wiki_convert(&tail, 0, 0); + style_footer(); + manifest_destroy(pEvent); +} + +/* +** WEBPAGE: eventedit +** URL: /eventedit?name=EVENTID +** +** Edit an event. If name is omitted, create a new event. +*/ +void eventedit_page(void){ + char *zTag; + int rid = 0; + Blob event; + const char *zEventId; + char *zHtmlPageName; + int n; + const char *z; + char *zBody = (char*)P("w"); + char *zETime = (char*)P("t"); + const char *zComment = P("c"); + const char *zTags = P("g"); + const char *zClr; + + if( zBody ){ + zBody = mprintf("%s", zBody); + } + login_check_credentials(); + zEventId = P("name"); + if( zEventId==0 ){ + zEventId = db_text(0, "SELECT lower(hex(randomblob(20)))"); + }else{ + int nEventId = strlen(zEventId); + if( nEventId!=40 || !validate16(zEventId, 40) ){ + fossil_redirect_home(); + return; + } + } + zTag = mprintf("event-%s", zEventId); + rid = db_int(0, + "SELECT rid FROM tagxref" + " WHERE tagid=(SELECT tagid FROM tag WHERE tagname=%Q)" + " ORDER BY mtime DESC", zTag + ); + free(zTag); + + /* Need both check-in and wiki-write or wiki-create privileges in order + ** to edit/create an event. + */ + if( !g.okWrite || (rid && !g.okWrWiki) || (!rid && !g.okNewWiki) ){ + login_needed(); + return; + } + + /* Figure out the color */ + if( rid ){ + zClr = db_text("", "SELECT bgcolor FROM event WHERE objid=%d", rid); + }else{ + zClr = ""; + } + zClr = PD("clr",zClr); + if( strcmp(zClr,"##")==0 ) zClr = PD("cclr",""); + + + /* If editing an existing event, extract the key fields to use as + ** a starting point for the edit. + */ + if( rid && (zBody==0 || zETime==0 || zComment==0 || zTags==0) ){ + Manifest *pEvent; + pEvent = manifest_get(rid, CFTYPE_EVENT); + if( pEvent && pEvent->type==CFTYPE_EVENT ){ + if( zBody==0 ) zBody = pEvent->zWiki; + if( zETime==0 ){ + zETime = db_text(0, "SELECT datetime(%.17g)", pEvent->rEventDate); + } + if( zComment==0 ) zComment = pEvent->zComment; + } + if( zTags==0 ){ + zTags = db_text(0, + "SELECT group_concat(substr(tagname,5),', ')" + " FROM tagxref, tag" + " WHERE tagxref.rid=%d" + " AND tagxref.tagid=tag.tagid" + " AND tag.tagname GLOB 'sym-*'", + rid + ); + } + } + zETime = db_text(0, "SELECT coalesce(datetime(%Q),datetime('now'))", zETime); + if( P("submit")!=0 && (zBody!=0 && zComment!=0) ){ + char *zDate; + Blob cksum; + int nrid; + blob_zero(&event); + db_begin_transaction(); + login_verify_csrf_secret(); + blob_appendf(&event, "C %F\n", zComment); + zDate = db_text(0, "SELECT datetime('now')"); + zDate[10] = 'T'; + blob_appendf(&event, "D %s\n", zDate); + free(zDate); + zETime[10] = 'T'; + blob_appendf(&event, "E %s %s\n", zETime, zEventId); + zETime[10] = ' '; + if( rid ){ + char *zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rid); + blob_appendf(&event, "P %s\n", zUuid); + free(zUuid); + } + if( zClr && zClr[0] ){ + blob_appendf(&event, "T +bgcolor * %F\n", zClr); + } + if( zTags && zTags[0] ){ + Blob tags, one; + int i, j; + Stmt q; + char *zBlob; + + /* Load the tags string into a blob */ + blob_zero(&tags); + blob_append(&tags, zTags, -1); + + /* Collapse all sequences of whitespace and "," characters into + ** a single space character */ + zBlob = blob_str(&tags); + for(i=j=0; zBlob[i]; i++, j++){ + if( fossil_isspace(zBlob[i]) || zBlob[i]==',' ){ + while( fossil_isspace(zBlob[i+1]) ){ i++; } + zBlob[j] = ' '; + }else{ + zBlob[j] = zBlob[i]; + } + } + blob_resize(&tags, j); + + /* Parse out each tag and load it into a temporary table for sorting */ + db_multi_exec("CREATE TEMP TABLE newtags(x);"); + while( blob_token(&tags, &one) ){ + db_multi_exec("INSERT INTO newtags VALUES(%B)", &one); + } + blob_reset(&tags); + + /* Extract the tags in sorted order and make an entry in the + ** artifact for each. */ + db_prepare(&q, "SELECT x FROM newtags ORDER BY x"); + while( db_step(&q)==SQLITE_ROW ){ + blob_appendf(&event, "T +sym-%F *\n", db_column_text(&q, 0)); + } + db_finalize(&q); + } + if( g.zLogin ){ + blob_appendf(&event, "U %F\n", g.zLogin); + } + blob_appendf(&event, "W %d\n%s\n", strlen(zBody), zBody); + md5sum_blob(&event, &cksum); + blob_appendf(&event, "Z %b\n", &cksum); + blob_reset(&cksum); + nrid = content_put(&event, 0, 0); + db_multi_exec("INSERT OR IGNORE INTO unsent VALUES(%d)", nrid); + manifest_crosslink(nrid, &event); + blob_reset(&event); + content_deltify(rid, nrid, 0); + db_end_transaction(0); + cgi_redirectf("event?name=%T", zEventId); + } + if( P("cancel")!=0 ){ + cgi_redirectf("event?name=%T", zEventId); + return; + } + if( zBody==0 ){ + zBody = mprintf("Event Text"); + } + zHtmlPageName = mprintf("Edit Event %S", zEventId); + style_header(zHtmlPageName); + if( P("preview")!=0 ){ + Blob title, tail, com; + @

Timeline comment preview:

+ @

+ @ + if( zClr && zClr[0] ){ + @
+ }else{ + @
+ } + blob_zero(&com); + blob_append(&com, zComment, -1); + wiki_convert(&com, 0, WIKI_INLINE); + @
+ @

+ @

Page content preview:

+ @

+ blob_zero(&event); + blob_append(&event, zBody, -1); + if( wiki_find_title(&event, &title, &tail) ){ + @
%h(blob_str(&title))
+ wiki_convert(&tail, 0, 0); + }else{ + wiki_convert(&event, 0, 0); + } + @

+ blob_reset(&event); + } + for(n=2, z=zBody; z[0]; z++){ + if( z[0]=='\n' ) n++; + } + if( n<20 ) n = 20; + if( n>40 ) n = 40; + @ + style_footer(); +} Index: src/file.c ================================================================== --- src/file.c +++ src/file.c @@ -23,12 +23,20 @@ #include #include "file.h" /* ** The file status information from the most recent stat() call. +** +** Use _stati64 rather than stat on windows, in order to handle files +** larger than 2GB. */ -static struct stat fileStat; +#if defined(_WIN32) && defined(__MSVCRT__) + static struct _stati64 fileStat; +# define stat _stati64 +#else + static struct stat fileStat; +#endif static int fileStatValid = 0; /* ** Fill in the fileStat variable for the file named zFilename. ** If zFilename==0, then use the previous value of fileStat if @@ -83,11 +91,14 @@ ** Return TRUE if the named file is an executable. Return false ** for directories, devices, fifos, symlinks, etc. */ int file_isexe(const char *zFilename){ if( getStat(zFilename) || !S_ISREG(fileStat.st_mode) ) return 0; -#ifdef __MINGW32__ +#if defined(_WIN32) +# if defined(__DMC__) || defined(_MSC_VER) +# define S_IXUSR _S_IEXEC +# endif return ((S_IXUSR)&fileStat.st_mode)!=0; #else return ((S_IXUSR|S_IXGRP|S_IXOTH)&fileStat.st_mode)!=0; #endif } @@ -145,11 +156,11 @@ /* ** Set or clear the execute bit on a file. */ void file_setexe(const char *zFilename, int onoff){ -#ifndef __MINGW32__ +#if !defined(_WIN32) struct stat buf; if( stat(zFilename, &buf)!=0 ) return; if( onoff ){ if( (buf.st_mode & 0111)!=0111 ){ chmod(zFilename, buf.st_mode | 0111); @@ -157,11 +168,11 @@ }else{ if( (buf.st_mode & 0111)!=0 ){ chmod(zFilename, buf.st_mode & ~0111); } } -#endif /* __MINGW32__ */ +#endif /* _WIN32 */ } /* ** Create the directory named in the argument, if it does not already ** exist. If forceFlag is 1, delete any prior non-directory object @@ -174,11 +185,11 @@ if( rc==2 ){ if( !forceFlag ) return 1; unlink(zName); } if( rc!=1 ){ -#ifdef __MINGW32__ +#if defined(_WIN32) return mkdir(zName); #else return mkdir(zName, 0755); #endif } @@ -231,11 +242,11 @@ ** Changes are made in-place. Return the new name length. */ int file_simplify_name(char *z, int n){ int i, j; if( n<0 ) n = strlen(z); -#ifdef __MINGW32__ +#if defined(_WIN32) for(i=0; i1 && z[n-1]=='/' ){ n--; } @@ -266,11 +277,11 @@ ** Remove all /./ path elements. ** Convert /A/../ to just / */ void file_canonical_name(const char *zOrigName, Blob *pOut){ if( zOrigName[0]=='/' -#ifdef __MINGW32__ +#if defined(_WIN32) || zOrigName[0]=='\\' || (strlen(zOrigName)>3 && zOrigName[1]==':' && (zOrigName[2]=='\\' || zOrigName[2]=='/')) #endif ){ @@ -288,21 +299,29 @@ blob_resize(pOut, file_simplify_name(blob_buffer(pOut), blob_size(pOut))); } /* ** COMMAND: test-canonical-name +** Usage: %fossil test-canonical-name FILENAME... ** ** Test the operation of the canonical name generator. +** Also test Fossil's ability to measure attributes of a file. */ void cmd_test_canonical_name(void){ int i; Blob x; blob_zero(&x); for(i=2; i%b(&title) blob_reset(&title); pGraph = graph_init(); @

- @ + @

while( db_step(&q)==SQLITE_ROW ){ const char *zDate = db_column_text(&q, 0); const char *zCom = db_column_text(&q, 1); const char *zUser = db_column_text(&q, 2); int fpid = db_column_int(&q, 3); @@ -157,22 +157,22 @@ if( zBr==0 ) zBr = "trunk"; gidx = graph_add_row(pGraph, frid, fpid>0 ? 1 : 0, &fpid, zBr, zBgClr); if( memcmp(zDate, zPrevDate, 10) ){ sprintf(zPrevDate, "%.10s", zDate); @ } memcpy(zTime, &zDate[11], 5); zTime[5] = 0; - @ - @ + @ if( zBgClr && zBgClr[0] ){ - @ + @ } db_finalize(&q); if( pGraph ){ graph_finish(pGraph, 1); if( pGraph->nErr ){ graph_free(pGraph); pGraph = 0; }else{ - @ } } @

timeline_output_graph_javascript(pGraph); style_footer(); } Index: src/graph.c ================================================================== --- src/graph.c +++ src/graph.c @@ -21,12 +21,12 @@ #include "graph.h" #include #if INTERFACE -#define GR_MAX_PARENT 10 -#define GR_MAX_RAIL 32 +#define GR_MAX_PARENT 10 /* Most parents for any one node */ +#define GR_MAX_RAIL 32 /* Max number of "rails" to display */ /* The graph appears vertically beside a timeline. Each row in the ** timeline corresponds to a row in the graph. */ struct GraphRow { @@ -38,11 +38,12 @@ GraphRow *pNext; /* Next row down in the list of all rows */ GraphRow *pPrev; /* Previous row */ int idx; /* Row index. First is 1. 0 used for "none" */ - u8 isLeaf; /* True if no direct child nodes */ + int idxTop; /* Direct descendent highest up on the graph */ + GraphRow *pChild; /* Child immediately above this node */ u8 isDup; /* True if this is duplicate of a prior entry */ int iRail; /* Which rail this check-in appears on. 0-based.*/ int aiRaiser[GR_MAX_RAIL]; /* Raisers from this node to a higher row. */ int bDescender; /* Raiser from bottom of graph to here. */ u32 mergeIn; /* Merge in from other rails */ @@ -53,31 +54,29 @@ }; /* Context while building a graph */ struct GraphContext { - int nErr; /* Number of errors encountered */ - int mxRail; /* Number of rails required to render the graph */ - GraphRow *pFirst; /* First row in the list */ - GraphRow *pLast; /* Last row in the list */ - int nBranch; /* Number of distinct branches */ - char **azBranch; /* Names of the branches */ + int nErr; /* Number of errors encountered */ + int mxRail; /* Number of rails required to render the graph */ + GraphRow *pFirst; /* First row in the list */ + GraphRow *pLast; /* Last row in the list */ + int nBranch; /* Number of distinct branches */ + char **azBranch; /* Names of the branches */ int nRow; /* Number of rows */ - int railMap[GR_MAX_RAIL]; /* Rail order mapping */ int nHash; /* Number of slots in apHash[] */ - GraphRow **apHash; /* Hash table of rows */ + GraphRow **apHash; /* Hash table of GraphRow objects. Key: rid */ }; #endif /* ** Malloc for zeroed space. Panic if unable to provide the ** requested space. */ void *safeMalloc(int nByte){ - void *p = malloc(nByte); - if( p==0 ) fossil_panic("out of memory"); + void *p = fossil_malloc(nByte); memset(p, 0, nByte); return p; } /* @@ -103,13 +102,13 @@ free(p->apHash); free(p); } /* -** Insert a row into the hash table. If there is already another -** row with the same rid, overwrite the prior entry if the overwrite -** flag is set. +** Insert a row into the hash table. pRow->rid is the key. Keys must +** be unique. If there is already another row with the same rid, +** overwrite the prior entry if and only if the overwrite flag is set. */ static void hashInsert(GraphContext *p, GraphRow *pRow, int overwrite){ int h; h = pRow->rid % p->nHash; while( p->apHash[h] && p->apHash[h]->rid!=pRow->rid ){ @@ -135,27 +134,30 @@ /* ** Return the canonical pointer for a given branch name. ** Multiple calls to this routine with equivalent strings ** will return the same pointer. +** +** The returned value is a pointer to a (readonly) string that +** has the useful property that strings can be checked for +** equality by comparing pointers. ** ** Note: also used for background color names. */ static char *persistBranchName(GraphContext *p, const char *zBranch){ int i; for(i=0; inBranch; i++){ if( strcmp(zBranch, p->azBranch[i])==0 ) return p->azBranch[i]; } p->nBranch++; - p->azBranch = realloc(p->azBranch, sizeof(char*)*p->nBranch); - if( p->azBranch==0 ) fossil_panic("out of memory"); + p->azBranch = fossil_realloc(p->azBranch, sizeof(char*)*p->nBranch); p->azBranch[p->nBranch-1] = mprintf("%s", zBranch); return p->azBranch[p->nBranch-1]; } /* -** Add a new row t the graph context. Rows are added from top to bottom. +** Add a new row to the graph context. Rows are added from top to bottom. */ int graph_add_row( GraphContext *p, /* The context to which the row is added */ int rid, /* RID for the check-in */ int nParent, /* Number of parents */ @@ -179,11 +181,11 @@ }else{ p->pLast->pNext = pRow; } p->pLast = pRow; p->nRow++; - pRow->idx = p->nRow; + pRow->idx = pRow->idxTop = p->nRow; return pRow->idx; } /* ** Return the index of a rail currently not in use for any row between @@ -219,16 +221,46 @@ } } if( iBestDist>1000 ) p->nErr++; return iBest; } + +/* +** Assign all children of node pBottom to the same rail as pBottom. +*/ +static void assignChildrenToRail(GraphRow *pBottom){ + int iRail = pBottom->iRail; + GraphRow *pCurrent; + GraphRow *pPrior; + u32 mask = 1<iRail = iRail; + pBottom->railInUse |= mask; + pPrior = pBottom; + for(pCurrent=pBottom->pChild; pCurrent; pCurrent=pCurrent->pChild){ + assert( pPrior->idx > pCurrent->idx ); + assert( pCurrent->iRail<0 ); + pCurrent->iRail = iRail; + pCurrent->railInUse |= mask; + pPrior->aiRaiser[iRail] = pCurrent->idx; + while( pPrior->idx > pCurrent->idx ){ + pPrior->railInUse |= mask; + pPrior = pPrior->pPrev; + assert( pPrior!=0 ); + } + if( pCurrent->pPrev ){ + pCurrent->pPrev->railInUse |= mask; + } + } +} + /* ** Compute the complete graph */ void graph_finish(GraphContext *p, int omitDescenders){ - GraphRow *pRow, *pDesc, *pDup, *pLoop; + GraphRow *pRow, *pDesc, *pDup, *pLoop, *pParent; int i; u32 mask; u32 inUse; int hasDup = 0; /* True if one or more isDup entries */ const char *zTrunk; @@ -248,11 +280,17 @@ } hashInsert(p, pRow, 1); } p->mxRail = -1; - /* Purge merge-parents that are out-of-graph + /* Purge merge-parents that are out-of-graph. + ** + ** Each node has one primary parent and zero or more "merge" parents. + ** A merge parent is a prior checkin from which changes were merged into + ** the current check-in. If a merge parent is not in the visible section + ** of this graph, then no arrows will be drawn for it, so remove it from + ** the aParent[] array. */ for(pRow=p->pFirst; pRow; pRow=pRow->pNext){ for(i=1; inParent; i++){ if( hashFind(p, pRow->aParent[i])==0 ){ pRow->aParent[i] = pRow->aParent[--pRow->nParent]; @@ -259,99 +297,106 @@ i--; } } } - /* Figure out which nodes have no direct children (children on - ** the same rail). Mark such nodes as isLeaf. + /* Find the pChild pointer for each node. + ** + ** The pChild points to node directly above on the same rail. + ** The pChild must be in the same branch. Leaf nodes have a NULL + ** pChild. + ** + ** In the case of a fork, choose the pChild that results in the + ** longest rail. */ - memset(p->apHash, 0, sizeof(p->apHash[0])*p->nHash); - for(pRow=p->pLast; pRow; pRow=pRow->pPrev) pRow->isLeaf = 1; - for(pRow=p->pLast; pRow; pRow=pRow->pPrev){ - GraphRow *pParent; - hashInsert(p, pRow, 0); - if( !pRow->isDup - && pRow->nParent>0 - && (pParent = hashFind(p, pRow->aParent[0]))!=0 - && pRow->zBranch==pParent->zBranch - ){ - pParent->isLeaf = 0; + for(pRow=p->pFirst; pRow; pRow=pRow->pNext){ + if( pRow->isDup ) continue; + if( pRow->nParent==0 ) continue; + pParent = hashFind(p, pRow->aParent[0]); + if( pParent==0 ) continue; + if( pParent->zBranch!=pRow->zBranch ) continue; + if( pParent->idx <= pRow->idx ) continue; + if( pRow->idxTop < pParent->idxTop ){ + pParent->pChild = pRow; + pParent->idxTop = pRow->idxTop; } } /* Identify rows where the primary parent is off screen. Assign ** each to a rail and draw descenders to the bottom of the screen. + ** + ** Strive to put the "trunk" branch on far left. */ zTrunk = persistBranchName(p, "trunk"); for(i=0; i<2; i++){ - for(pRow=p->pFirst; pRow; pRow=pRow->pNext){ + for(pRow=p->pLast; pRow; pRow=pRow->pPrev){ if( i==0 ){ if( pRow->zBranch!=zTrunk ) continue; }else { if( pRow->iRail>=0 ) continue; } if( pRow->nParent==0 || hashFind(p,pRow->aParent[0])==0 ){ if( omitDescenders ){ - pRow->iRail = findFreeRail(p, pRow->idx, pRow->idx, 0, 0); + pRow->iRail = findFreeRail(p, pRow->idxTop, pRow->idx, 0, 0); }else{ pRow->iRail = ++p->mxRail; } mask = 1<<(pRow->iRail); if( omitDescenders ){ if( pRow->pNext ) pRow->pNext->railInUse |= mask; - for(pDesc=pRow; pDesc; pDesc=pDesc->pPrev){ - pDesc->railInUse |= mask; - if( pDesc->zBranch==pRow->zBranch && pDesc->isLeaf ) break; - } }else{ pRow->bDescender = pRow->nParent>0; - for(pDesc=pRow; pDesc; pDesc=pDesc->pNext){ - pDesc->railInUse |= mask; + for(pLoop=pRow; pLoop; pLoop=pLoop->pNext){ + pLoop->railInUse |= mask; } } + assignChildrenToRail(pRow); } } } /* Assign rails to all rows that are still unassigned. - ** The first primary child of a row goes on the same rail as - ** that row. */ inUse = (1<<(p->mxRail+1))-1; for(pRow=p->pLast; pRow; pRow=pRow->pPrev){ int parentRid; - if( pRow->iRail>=0 ) continue; + + if( pRow->iRail>=0 ){ + if( pRow->pChild==0 ) inUse &= ~(1<iRail); + continue; + } if( pRow->isDup ){ pRow->iRail = findFreeRail(p, pRow->idx, pRow->idx, inUse, 0); pDesc = pRow; + pParent = 0; }else{ assert( pRow->nParent>0 ); parentRid = pRow->aParent[0]; - pDesc = hashFind(p, parentRid); - if( pDesc==0 ){ + pParent = hashFind(p, parentRid); + if( pParent==0 ){ /* Time skew */ pRow->iRail = ++p->mxRail; pRow->railInUse = 1<iRail; continue; } - if( pDesc->aiRaiser[pDesc->iRail]==0 && pDesc->zBranch==pRow->zBranch ){ - pRow->iRail = pDesc->iRail; - }else{ - pRow->iRail = findFreeRail(p, 0, pDesc->idx, inUse, pDesc->iRail); - } - pDesc->aiRaiser[pRow->iRail] = pRow->idx; + pRow->iRail = findFreeRail(p, 0, pParent->idx, inUse, pParent->iRail); + pParent->aiRaiser[pRow->iRail] = pRow->idx; } mask = 1<iRail; - if( pRow->isLeaf ){ + if( pRow->pPrev ) pRow->pPrev->railInUse |= mask; + if( pRow->pNext ) pRow->pNext->railInUse |= mask; + if( pRow->pChild==0 ){ inUse &= ~mask; }else{ inUse |= mask; + assignChildrenToRail(pRow); } - for(pLoop=pRow; pLoop && pLoop!=pDesc; pLoop=pLoop->pNext){ - pLoop->railInUse |= mask; + if( pParent ){ + for(pLoop=pParent; pLoop && pLoop!=pRow; pLoop=pLoop->pPrev){ + pLoop->railInUse |= mask; + } } - pDesc->railInUse |= mask; } /* ** Insert merge rails and merge arrows */ @@ -364,13 +409,13 @@ int iTarget = (pRow->iRail + pDesc->iRail)/2; pDesc->mergeOut = findFreeRail(p, pRow->idx, pDesc->idx, 0, iTarget); pDesc->mergeUpto = pRow->idx; mask = 1<mergeOut; pDesc->railInUse |= mask; - for(pDesc=pRow->pNext; pDesc && pDesc->rid!=parentRid; - pDesc=pDesc->pNext){ - pDesc->railInUse |= mask; + for(pLoop=pRow->pNext; pLoop && pLoop->rid!=parentRid; + pLoop=pLoop->pNext){ + pLoop->railInUse |= mask; } } pRow->mergeIn |= 1<mergeOut; } } @@ -398,12 +443,11 @@ } /* ** Find the maximum rail number. */ - for(i=0; irailMap[i] = i; p->mxRail = 0; for(pRow=p->pFirst; pRow; pRow=pRow->pNext){ if( pRow->iRail>p->mxRail ) p->mxRail = pRow->iRail; if( pRow->mergeOut>p->mxRail ) p->mxRail = pRow->mergeOut; } } Index: src/http.c ================================================================== --- src/http.c +++ src/http.c @@ -42,10 +42,13 @@ blob_zero(pLogin); if( g.urlUser==0 || strcmp(g.urlUser, "anonymous")==0 ){ return; /* If no login card for users "nobody" and "anonymous" */ } + if( g.urlIsSsh ){ + return; /* If no login card for SSH: */ + } blob_zero(&nonce); blob_zero(&pw); sha1sum_blob(pPayload, &nonce); blob_copy(&pw, &nonce); zLogin = g.urlUser; @@ -57,11 +60,11 @@ zPw = 0; }else{ /* Password failure while doing a sync from the command-line interface */ url_prompt_for_password(); zPw = g.urlPasswd; - if( !g.dontKeepUrl ) db_set("last-sync-pw", zPw, 0); + if( !g.dontKeepUrl ) db_set("last-sync-pw", obscure(zPw), 0); } /* The login card wants the SHA1 hash of the password, so convert the ** password to its SHA1 hash it it isn't already a SHA1 hash. ** @@ -100,11 +103,11 @@ if( i>0 && g.urlPath[i-1]=='/' ){ zSep = ""; }else{ zSep = "/"; } - blob_appendf(pHdr, "POST %s%sxfer HTTP/1.0\r\n", g.urlPath, zSep); + blob_appendf(pHdr, "POST %s%sxfer/xfer HTTP/1.0\r\n", g.urlPath, zSep); if( g.urlProxyAuth ){ blob_appendf(pHdr, "Proxy-Authorization: %s\n", g.urlProxyAuth); } blob_appendf(pHdr, "Host: %s\r\n", g.urlHostname); blob_appendf(pHdr, "User-Agent: Fossil/" MANIFEST_VERSION "\r\n"); @@ -134,10 +137,11 @@ int iLength; /* Length of the reply payload */ int rc; /* Result code */ int iHttpVersion; /* Which version of HTTP protocol server uses */ char *zLine; /* A single line of the reply header */ int i; /* Loop counter */ + int isError = 0; /* True if the reply is an error message */ if( transport_open() ){ fossil_fatal(transport_errmsg()); } @@ -190,10 +194,11 @@ ** Read and interpret the server reply */ closeConnection = 1; iLength = -1; while( (zLine = transport_receive_line())!=0 && zLine[0]!=0 ){ + /* printf("[%s]\n", zLine); fflush(stdout); */ if( strncasecmp(zLine, "http/1.", 7)==0 ){ if( sscanf(zLine, "HTTP/1.%d %d", &iHttpVersion, &rc)!=2 ) goto write_err; if( rc!=200 && rc!=302 ){ int ii; for(ii=7; zLine[ii] && zLine[ii]!=' '; ii++){} @@ -205,15 +210,15 @@ closeConnection = 1; }else{ closeConnection = 0; } }else if( strncasecmp(zLine, "content-length:", 15)==0 ){ - for(i=15; isspace(zLine[i]); i++){} + for(i=15; fossil_isspace(zLine[i]); i++){} iLength = atoi(&zLine[i]); }else if( strncasecmp(zLine, "connection:", 11)==0 ){ char c; - for(i=11; isspace(zLine[i]); i++){} + for(i=11; fossil_isspace(zLine[i]); i++){} c = zLine[i]; if( c=='c' || c=='C' ){ closeConnection = 1; }else if( c=='k' || c=='K' ){ closeConnection = 0; @@ -221,16 +226,21 @@ }else if( rc==302 && strncasecmp(zLine, "location:", 9)==0 ){ int i, j; for(i=9; zLine[i] && zLine[i]==' '; i++){} if( zLine[i]==0 ) fossil_fatal("malformed redirect: %s", zLine); j = strlen(zLine) - 1; - if( j>4 && strcmp(&zLine[j-4],"/xfer")==0 ) zLine[j-4] = 0; + while( j>4 && strcmp(&zLine[j-4],"/xfer")==0 ){ + j -= 4; + zLine[j] = 0; + } fossil_print("redirect to %s\n", &zLine[i]); url_parse(&zLine[i]); transport_close(); http_exchange(pSend, pReply, useLogin); return; + }else if( strncasecmp(zLine, "content-type: text/html", 23)==0 ){ + isError = 1; } } if( rc!=200 ){ fossil_fatal("\"location:\" missing from 302 redirect reply"); goto write_err; @@ -245,12 +255,26 @@ } blob_zero(pReply); blob_resize(pReply, iLength); iLength = transport_receive(blob_buffer(pReply), iLength); blob_resize(pReply, iLength); + if( isError ){ + char *z; + int i, j; + z = blob_str(pReply); + for(i=j=0; z[i]; i++, j++){ + if( z[i]=='<' ){ + while( z[i] && z[i]!='>' ) i++; + if( z[i]==0 ) break; + } + z[j] = z[i]; + } + z[j] = 0; + fossil_fatal("server sends error: %s", z); + } if( g.fHttpTrace ){ - printf("HTTP RECEIVE:\n%s\n=======================\n", blob_str(pReply)); + /*printf("HTTP RECEIVE:\n%s\n=======================\n",blob_str(pReply));*/ }else{ blob_uncompress(pReply, pReply); } /* Index: src/http_socket.c ================================================================== --- src/http_socket.c +++ src/http_socket.c @@ -26,13 +26,16 @@ ** are handled different on Unix and windows. */ #include "config.h" #include "http_socket.h" -#ifdef __MINGW32__ -# include -# include +#if defined(_WIN32) +# include /* for Sleep once server works again */ +# define sleep Sleep /* windows does not have sleep, but Sleep */ +# if defined(__MINGW32__) +# include +# endif #else # include # include # include # include @@ -45,11 +48,11 @@ ** There can only be a single socket connection open at a time. ** State information about that socket is stored in the following ** local variables: */ static int socketIsInit = 0; /* True after global initialization */ -#ifdef __MINGW32__ +#if defined(_WIN32) static WSADATA socketInfo; /* Windows socket initialize data */ #endif static int iSocket = -1; /* The socket on which we talk to the server */ static char *socketErrMsg = 0; /* Text of most recent socket error */ @@ -84,11 +87,11 @@ ** Call this routine once before any other use of the socket interface. ** This routine does initial configuration of the socket module. */ void socket_global_init(void){ if( socketIsInit==0 ){ -#ifdef __MINGW32__ +#if defined(_WIN32) if( WSAStartup(MAKEWORD(2,0), &socketInfo)!=0 ){ fossil_panic("can't initialize winsock"); } #endif socketIsInit = 1; @@ -99,11 +102,11 @@ ** Call this routine to shutdown the socket module prior to program ** exit. */ void socket_global_shutdown(void){ if( socketIsInit ){ -#ifdef __MINGW32__ +#if defined(_WIN32) WSACleanup(); #endif socket_clear_errmsg(); socketIsInit = 0; } @@ -113,11 +116,11 @@ ** Close the currently open socket. If no socket is open, this routine ** is a no-op. */ void socket_close(void){ if( iSocket>=0 ){ -#ifdef __MINGW32__ +#if defined(_WIN32) closesocket(iSocket); #else close(iSocket); #endif iSocket = -1; @@ -171,11 +174,11 @@ if( connect(iSocket,(struct sockaddr*)&addr,sizeof(addr))<0 ){ socket_set_errmsg("cannot connect to host %s:%d", g.urlName, g.urlPort); socket_close(); return 1; } -#ifndef __MINGW32__ +#if !defined(_WIN32) signal(SIGPIPE, SIG_IGN); #endif return 0; } Index: src/http_ssl.c ================================================================== --- src/http_ssl.c +++ src/http_ssl.c @@ -90,10 +90,11 @@ SSL_library_init(); SSL_load_error_strings(); ERR_load_BIO_strings(); OpenSSL_add_all_algorithms(); sslCtx = SSL_CTX_new(SSLv23_client_method()); + X509_STORE_set_default_paths(SSL_CTX_get_cert_store(sslCtx)); sslIsInit = 1; } } /* @@ -128,11 +129,11 @@ ** Return the number of errors. */ int ssl_open(void){ X509 *cert; int hasSavedCertificate = 0; - +char *connStr ; ssl_global_init(); /* Get certificate for current server from global config and * (if we have it in config) add it to certificate store. */ @@ -150,11 +151,11 @@ ssl_set_errmsg("SSL: cannot open SSL (%s)", ERR_reason_error_string(ERR_get_error())); return 1; } - char *connStr = mprintf("%s:%d", g.urlName, g.urlPort); + connStr = mprintf("%s:%d", g.urlName, g.urlPort); BIO_set_conn_hostname(iBio, connStr); free(connStr); if( BIO_do_connect(iBio)<=0 ){ ssl_set_errmsg("SSL: cannot connect to host %s:%d (%s)", @@ -178,29 +179,29 @@ return 1; } if( SSL_get_verify_result(ssl) != X509_V_OK ){ char *desc, *prompt; + char *warning = ""; + Blob ans; BIO *mem; mem = BIO_new(BIO_s_mem()); X509_NAME_print_ex(mem, X509_get_subject_name(cert), 2, XN_FLAG_MULTILINE); BIO_puts(mem, "\n\nIssued By:\n\n"); X509_NAME_print_ex(mem, X509_get_issuer_name(cert), 2, XN_FLAG_MULTILINE); BIO_write(mem, "", 1); // null-terminate mem buffer BIO_get_mem_data(mem, &desc); - char *warning = ""; if( hasSavedCertificate ){ warning = "WARNING: Certificate doesn't match the " "saved certificate for this host!"; } prompt = mprintf("\nUnknown SSL certificate:\n\n%s\n\n%s\n" "Accept certificate [a=always/y/N]? ", desc, warning); BIO_free(mem); - Blob ans; prompt_user(prompt, &ans); free(prompt); if( blob_str(&ans)[0]!='y' && blob_str(&ans)[0]!='a' ) { X509_free(cert); ssl_set_errmsg("SSL certificate declined"); Index: src/http_transport.c ================================================================== --- src/http_transport.c +++ src/http_transport.c @@ -38,10 +38,19 @@ char *zOutFile; /* Name of outbound file for FILE: */ char *zInFile; /* Name of inbound file for FILE: */ } transport = { 0, 0, 0, 0, 0, 0, 0 }; + +/* +** Information about the connection to the SSH subprocess when +** using the ssh:// sync method. +*/ +static int sshPid; /* Process id of ssh subprocess */ +static int sshIn; /* From ssh subprocess to this process */ +static FILE *sshOut; /* From this to ssh subprocess */ + /* ** Return the current transport error message. */ const char *transport_errmsg(void){ @@ -63,10 +72,102 @@ if( resetFlag ){ transport.nSent = 0; transport.nRcvd = 0; } } + +/* +** Read text from sshIn. Zero-terminate and remove trailing +** whitespace. +*/ +static void sshin_read(char *zBuf, int szBuf){ + int got; + zBuf[0] = 0; + got = read(sshIn, zBuf, szBuf-1); + while( got>=0 ){ + zBuf[got] = 0; + if( got==0 || !fossil_isspace(zBuf[got-1]) ) break; + got--; + } +} + +/* +** Default SSH command +*/ +#ifdef __MINGW32__ +static char zDefaultSshCmd[] = "ssh -T"; +#else +static char zDefaultSshCmd[] = "ssh -e none -T"; +#endif + +/* +** Global initialization of the transport layer +*/ +void transport_global_startup(void){ + if( g.urlIsSsh ){ + /* Only SSH requires a global initialization. For SSH we need to create + ** and run an SSH command to talk to the remote machine. + */ + const char *zSsh; /* The base SSH command */ + Blob zCmd; /* The SSH command */ + char *zHost; /* The host name to contact */ + char zIn[200]; /* An input line received back from remote */ + + zSsh = db_get("ssh-command", zDefaultSshCmd); + blob_init(&zCmd, zSsh, -1); + if( g.urlPort!=g.urlDfltPort ){ +#ifdef __MINGW32__ + blob_appendf(&zCmd, " -P %d", g.urlPort); +#else + blob_appendf(&zCmd, " -p %d", g.urlPort); +#endif + } + if( g.urlUser && g.urlUser[0] ){ + zHost = mprintf("%s@%s", g.urlUser, g.urlName); +#ifdef __MINGW32__ + /* Only win32 (and specifically PLINK.EXE support the -pw option */ + if( g.urlPasswd && g.urlPasswd[0] ){ + Blob pw; + blob_zero(&pw); + if( g.urlPasswd[0]=='*' ){ + char *zPrompt; + zPrompt = mprintf("Password for [%s]: ", zHost); + prompt_for_password(zPrompt, &pw, 0); + free(zPrompt); + }else{ + blob_init(&pw, g.urlPasswd, -1); + } + blob_append(&zCmd, " -pw ", -1); + shell_escape(&zCmd, blob_str(&pw)); + blob_reset(&pw); + } +#endif + }else{ + zHost = mprintf("%s", g.urlName); + } + blob_append(&zCmd, " ", 1); + shell_escape(&zCmd, zHost); + free(zHost); + /* printf("%s\n", blob_str(&zCmd)); */ + popen2(blob_str(&zCmd), &sshIn, &sshOut, &sshPid); + if( sshPid==0 ){ + fossil_fatal("cannot start ssh tunnel using [%b]", &zCmd); + } + blob_reset(&zCmd); + + /* Send an "echo" command to the other side to make sure that the + ** connection is up and working. + */ + fprintf(sshOut, "echo test\n"); + fflush(sshOut); + sshin_read(zIn, sizeof(zIn)); + if( memcmp(zIn, "test", 4)!=0 ){ + pclose2(sshIn, sshOut, sshPid); + fossil_fatal("ssh connection failed: [%s]", zIn); + } + } +} /* ** Open a connection to the server. The server is defined by the following ** global variables: ** @@ -77,11 +178,21 @@ ** Return the number of errors. */ int transport_open(void){ int rc = 0; if( transport.isOpen==0 ){ - if( g.urlIsHttps ){ + if( g.urlIsSsh ){ + Blob cmd; + blob_zero(&cmd); + shell_escape(&cmd, g.urlFossil); + blob_append(&cmd, " test-http ", -1); + shell_escape(&cmd, g.urlPath); + /* fprintf(stdout, "%s\n", blob_str(&cmd)); */ + fprintf(sshOut, "%s\n", blob_str(&cmd)); + fflush(sshOut); + blob_reset(&cmd); + }else if( g.urlIsHttps ){ #ifdef FOSSIL_ENABLE_SSL rc = ssl_open(); if( rc==0 ) transport.isOpen = 1; #else socket_set_errmsg("HTTPS: Fossil has been compiled without SSL support"); @@ -115,11 +226,13 @@ free(transport.pBuf); transport.pBuf = 0; transport.nAlloc = 0; transport.nUsed = 0; transport.iCursor = 0; - if( g.urlIsHttps ){ + if( g.urlIsSsh ){ + /* No-op */ + }else if( g.urlIsHttps ){ #ifdef FOSSIL_ENABLE_SSL ssl_close(); #endif }else if( g.urlIsFile ){ if( transport.pFile ){ @@ -142,11 +255,16 @@ */ void transport_send(Blob *toSend){ char *z = blob_buffer(toSend); int n = blob_size(toSend); transport.nSent += n; - if( g.urlIsHttps ){ + if( g.urlIsSsh ){ + int sent; + sent = fwrite(z, 1, n, sshOut); + fflush(sshOut); + /* printf("sent %d of %d bytes\n", sent, n); fflush(stdout); */ + }else if( g.urlIsHttps ){ #ifdef FOSSIL_ENABLE_SSL int sent; while( n>0 ){ sent = ssl_send(0, z, n); /* printf("Sent %d of %d bytes\n", sent, n); fflush(stdout); */ @@ -170,17 +288,19 @@ /* ** This routine is called when the outbound message is complete and ** it is time to being recieving a reply. */ void transport_flip(void){ - if( g.urlIsFile ){ + if( g.urlIsSsh ){ + fprintf(sshOut, "\n\n"); + }else if( g.urlIsFile ){ char *zCmd; fclose(transport.pFile); zCmd = mprintf("\"%s\" http \"%s\" \"%s\" \"%s\" 127.0.0.1", g.argv[0], g.urlName, transport.zOutFile, transport.zInFile ); - portable_system(zCmd); + fossil_system(zCmd); free(zCmd); transport.pFile = fopen(transport.zInFile, "rb"); } } @@ -191,10 +311,41 @@ void transport_rewind(void){ if( g.urlIsFile ){ transport_close(); } } + +/* +** Read N bytes of content directly from the wire and write into +** the buffer. +*/ +static int transport_fetch(char *zBuf, int N){ + int got; + if( sshIn ){ + int x; + int wanted = N; + got = 0; + while( wanted>0 ){ + x = read(sshIn, &zBuf[got], wanted); + if( x<=0 ) break; + got += x; + wanted -= x; + } + }else if( g.urlIsHttps ){ + #ifdef FOSSIL_ENABLE_SSL + got = ssl_receive(0, zBuf, N); + #else + got = 0; + #endif + }else if( g.urlIsFile ){ + got = fread(zBuf, 1, N, transport.pFile); + }else{ + got = socket_receive(0, zBuf, N); + } + /* printf("received %d of %d bytes\n", got, N); fflush(stdout); */ + return got; +} /* ** Read N bytes of content from the wire and store in the supplied buffer. ** Return the number of bytes actually received. */ @@ -201,10 +352,11 @@ int transport_receive(char *zBuf, int N){ int onHand; /* Bytes current held in the transport buffer */ int nByte = 0; /* Bytes of content received */ onHand = transport.nUsed - transport.iCursor; + /* printf("request %d with %d on hand\n", N, onHand); fflush(stdout); */ if( onHand>0 ){ int toMove = onHand; if( toMove>N ) toMove = N; /* printf("bytes on hand: %d of %d\n", toMove, N); fflush(stdout); */ memcpy(zBuf, &transport.pBuf[transport.iCursor], toMove); @@ -216,24 +368,11 @@ N -= toMove; zBuf += toMove; nByte += toMove; } if( N>0 ){ - int got; - if( g.urlIsHttps ){ - #ifdef FOSSIL_ENABLE_SSL - got = ssl_receive(0, zBuf, N); - /* printf("received %d of %d bytes\n", got, N); fflush(stdout); */ - #else - got = 0; - #endif - }else if( g.urlIsFile ){ - got = fread(zBuf, 1, N, transport.pFile); - }else{ - got = socket_receive(0, zBuf, N); - /* printf("received %d of %d bytes\n", got, N); fflush(stdout); */ - } + int got = transport_fetch(zBuf, N); if( got>0 ){ nByte += got; transport.nRcvd += got; } } @@ -247,12 +386,11 @@ */ static void transport_load_buffer(int N){ int i, j; if( transport.nAlloc==0 ){ transport.nAlloc = N; - transport.pBuf = malloc( N ); - if( transport.pBuf==0 ) fossil_panic("out of memory"); + transport.pBuf = fossil_malloc( N ); transport.iCursor = 0; transport.nUsed = 0; } if( transport.iCursor>0 ){ for(i=0, j=transport.iCursor; j transport.nAlloc ){ char *pNew; transport.nAlloc = transport.nUsed + N; - pNew = realloc(transport.pBuf, transport.nAlloc); - if( pNew==0 ) fossil_panic("out of memory"); + pNew = fossil_realloc(transport.pBuf, transport.nAlloc); transport.pBuf = pNew; } if( N>0 ){ - i = transport_receive(&transport.pBuf[transport.nUsed], N); + i = transport_fetch(&transport.pBuf[transport.nUsed], N); if( i>0 ){ transport.nUsed += i; } } } @@ -300,11 +437,11 @@ break; } } if( transport.pBuf[i]=='\n' ){ transport.iCursor = i+1; - while( i>=iStart && isspace(transport.pBuf[i]) ){ + while( i>=iStart && fossil_isspace(transport.pBuf[i]) ){ transport.pBuf[i] = 0; i--; } break; } @@ -313,13 +450,19 @@ /* printf("Got line: [%s]\n", &transport.pBuf[iStart]); */ return &transport.pBuf[iStart]; } void transport_global_shutdown(void){ + if( g.urlIsSsh && sshPid ){ + printf("Closing SSH tunnel: "); + fflush(stdout); + pclose2(sshIn, sshOut, sshPid); + sshPid = 0; + } if( g.urlIsHttps ){ #ifdef FOSSIL_ENABLE_SSL ssl_global_shutdown(); #endif }else{ socket_global_shutdown(); } } Index: src/info.c ================================================================== --- src/info.c +++ src/info.c @@ -65,10 +65,16 @@ ); /* 01234567890123 */ printf("%-13s %s %s\n", zUuidName, zUuid, zDate ? zDate : ""); free(zUuid); free(zDate); + } + if( zUuid && showComment ){ + zComment = db_text(0, + "SELECT coalesce(ecomment,comment) || ' (user: ' || coalesce(euser,user,'?') || ')' FROM event WHERE objid=%d", + rid + ); } db_prepare(&q, "SELECT uuid, pid FROM plink JOIN blob ON pid=rid " " WHERE cid=%d", rid); while( db_step(&q)==SQLITE_ROW ){ const char *zUuid = db_column_text(&q, 0); @@ -96,11 +102,12 @@ if( zTags && zTags[0] ){ printf("tags: %s\n", zTags); } free(zTags); if( zComment ){ - printf("comment:\n%s\n", zComment); + printf("comment: "); + comment_print(zComment, 14, 79); free(zComment); } } @@ -135,11 +142,11 @@ /* 012345678901234 */ db_record_repository_filename(0); printf("project-name: %s\n", db_get("project-name", "")); printf("repository: %s\n", db_lget("repository", "")); printf("local-root: %s\n", g.zLocalRoot); -#ifdef __MINGW32__ +#if defined(_WIN32) if( g.zHome ){ printf("user-home: %s\n", g.zHome); } #endif printf("project-code: %s\n", db_get("project-code", "")); @@ -187,15 +194,15 @@ @

Tags And Properties

if( tagtype==0 ){ - @ ~~%h(zTagname)~~ cancelled + @ %h(zTagname) cancelled }else if( zValue ){ - @ %h(zTagname)=%h(zValue) + @ %h(zTagname)=%h(zValue) }else { - @ %h(zTagname) + @ %h(zTagname) } if( tagtype==2 ){ if( zOrigUuid && zOrigUuid[0] ){ @ inherited from hyperlink_to_uuid(zOrigUuid); @@ -215,10 +222,11 @@ } hyperlink_to_uuid(zSrcUuid); @ on hyperlink_to_date(zDate,0); } + @

} @@ -226,16 +234,28 @@ /* ** Append the difference between two RIDs to the output */ -static void append_diff(int fromid, int toid){ +static void append_diff(const char *zFrom, const char *zTo){ + int fromid; + int toid; Blob from, to, out; - content_get(fromid, &from); - content_get(toid, &to); + if( zFrom ){ + fromid = uuid_to_rid(zFrom, 0); + content_get(fromid, &from); + }else{ + blob_zero(&from); + } + if( zTo ){ + toid = uuid_to_rid(zTo, 0); + content_get(toid, &to); + }else{ + blob_zero(&to); + } blob_zero(&out); - text_diff(&from, &to, &out, 5); + text_diff(&from, &to, &out, 5, 1); @ %h(blob_str(&out)) blob_reset(&from); blob_reset(&to); blob_reset(&out); } @@ -256,30 +276,36 @@ }else if( zOld==0 ){ @

Added %h(zName)

}else{ @

Changes to %h(zName)

} - }else if( zOld && zNew ){ - @

Modified %h(zName) - @ from [%S(zOld)] - @ to [%S(zNew)]. - if( !showDiff ){ - @ - @ [diff] + if( showDiff ){ + @

+      append_diff(zOld, zNew);
+      @

+ } + }else{ + if( zOld && zNew ){ + @

Modified %h(zName) + @ from [%S(zOld)] + @ to [%S(zNew)]. + }else if( zOld ){ + @

Deleted %h(zName) + @ version [%S(zOld)] }else{ - int rid1 = uuid_to_rid(zOld, 0); - int rid2 = uuid_to_rid(zNew, 0); + @

Added %h(zName) + @ version [%S(zNew)] + } + if( showDiff ){ @

-      append_diff(rid1, rid2);
+      append_diff(zOld, zNew);
       @

+ }else if( zOld && zNew ){ + @ + @ [diff] } - }else if( zOld ){ - @

Deleted %h(zName) - @ version [%S(zOld)]

- }else{ - @

Added %h(zName) - @ version [%S(zNew)]

+ @

} } /* @@ -340,11 +366,11 @@ TAG_COMMENT, rid); zUser = db_column_text(&q, 2); zComment = db_column_text(&q, 3); zDate = db_column_text(&q,1); @

Overview

- @

+ @
@ @@ -383,13 +409,13 @@ db_finalize(&q); } if( g.okHistory ){ const char *zProjName = db_get("project-name", "unnamed"); @ @ @ @ } - @
SHA1 Hash: %s(zUuid) if( g.okSetup ){ @ (Record ID: %d(rid)) } @
Timelines: - @ ancestors - @ | descendants - @ | both + @ ancestors + @ | descendants + @ | both db_prepare(&q, "SELECT substr(tag.tagname,5) FROM tagxref, tag " " WHERE rid=%d AND tagtype>0 " " AND tag.tagid=tagxref.tagid " " AND +tag.tagname GLOB 'sym-*'", rid); while( db_step(&q)==SQLITE_ROW ){ @@ -399,20 +425,22 @@ db_finalize(&q); @
Other Links: @ files - @ | - @ ZIP archive + if( g.okZip ){ + @ | + @ ZIP archive + } @ | manifest if( g.okWrite ){ @ | edit } @

SHA1 Hash:	%s(zUuid) if( g.okSetup ){ @ (Record ID: %d(rid)) } @
Timelines:	- @ ancestors - @ \| descendants - @ \| both + @ ancestors + @ \| descendants + @ \| both db_prepare(&q, "SELECT substr(tag.tagname,5) FROM tagxref, tag " " WHERE rid=%d AND tagtype>0 " " AND tag.tagid=tagxref.tagid " " AND +tag.tagname GLOB 'sym-*'", rid); while( db_step(&q)==SQLITE_ROW ){ @@ -399,20 +425,22 @@ db_finalize(&q); @
Other Links:	@ files - @ \| - @ ZIP archive + if( g.okZip ){ + @ \| + @ ZIP archive + } @ \| manifest if( g.okWrite ){ @ \| edit } @

+ @ }else{ style_header("Check-in Information"); login_anonymous_available(); } db_finalize(&q); @@ -515,24 +543,20 @@ rid = 0; } db_finalize(&q); showTags(rid, "wiki-*"); if( rid ){ - Blob content; - Manifest m; - memset(&m, 0, sizeof(m)); - blob_zero(&m.content); - content_get(rid, &content); - manifest_parse(&m, &content); - if( m.type==CFTYPE_WIKI ){ + Manifest *pWiki; + pWiki = manifest_get(rid, CFTYPE_WIKI); + if( pWiki ){ Blob wiki; - blob_init(&wiki, m.zWiki, -1); + blob_init(&wiki, pWiki->zWiki, -1); @

Content

wiki_convert(&wiki, 0, 0); blob_reset(&wiki); } - manifest_clear(&m); + manifest_destroy(pWiki); } style_footer(); } /* @@ -552,26 +576,23 @@ /* ** Find an checkin based on query parameter zParam and parse its ** manifest. Return the number of errors. */ -static int vdiff_parse_manifest(const char *zParam, int *pRid, Manifest *pM){ +static Manifest *vdiff_parse_manifest(const char *zParam, int *pRid){ int rid; - Blob content; *pRid = rid = name_to_rid_www(zParam); if( rid==0 ){ webpage_error("Missing \"%s\" query parameter.", zParam); - return 1; + return 0; } if( !is_a_version(rid) ){ webpage_error("Artifact %s is not a checkin.", P(zParam)); - return 1; + return 0; } - content_get(rid, &content); - manifest_parse(pM, &content); - return 0; + return manifest_get(rid, CFTYPE_MANIFEST); } /* ** Output a description of a check-in */ @@ -600,66 +621,71 @@ } /* ** WEBPAGE: vdiff -** URL: /vdiff?from=UUID&to=UUID&detail=BOOLEAN +** URL: /vdiff?from=UUID&to=UUID&detail=BOOLEAN ** ** Show all differences between two checkins. */ void vdiff_page(void){ int ridFrom, ridTo; int showDetail = 0; - int iFrom, iTo; - Manifest mFrom, mTo; + Manifest *pFrom, *pTo; + ManifestFile *pFileFrom, *pFileTo; login_check_credentials(); if( !g.okRead ){ login_needed(); return; } login_anonymous_available(); - if( vdiff_parse_manifest("from", &ridFrom, &mFrom) ) return; - if( vdiff_parse_manifest("to", &ridTo, &mTo) ) return; + pFrom = vdiff_parse_manifest("from", &ridFrom); + if( pFrom==0 ) return; + pTo = vdiff_parse_manifest("to", &ridTo); + if( pTo==0 ) return; showDetail = atoi(PD("detail","0")); style_header("Check-in Differences"); @

checkin_description(ridFrom); @

checkin_description(ridTo); - @

+ @

- iFrom = iTo = 0; - while( iFrom=mFrom.nFile ){ + if( pFileFrom==0 ){ cmp = +1; - }else if( iTo>=mTo.nFile ){ + }else if( pFileTo==0 ){ cmp = -1; }else{ - cmp = strcmp(mFrom.aFile[iFrom].zName, mTo.aFile[iTo].zName); + cmp = strcmp(pFileFrom->zName, pFileTo->zName); } if( cmp<0 ){ - append_file_change_line(mFrom.aFile[iFrom].zName, - mFrom.aFile[iFrom].zUuid, 0, 0); - iFrom++; + append_file_change_line(pFileFrom->zName, + pFileFrom->zUuid, 0, 0); + pFileFrom = manifest_file_next(pFrom, 0); }else if( cmp>0 ){ - append_file_change_line(mTo.aFile[iTo].zName, - 0, mTo.aFile[iTo].zUuid, 0); - iTo++; - }else if( strcmp(mFrom.aFile[iFrom].zUuid, mTo.aFile[iTo].zUuid)==0 ){ + append_file_change_line(pFileTo->zName, + 0, pFileTo->zUuid, 0); + pFileTo = manifest_file_next(pTo, 0); + }else if( strcmp(pFileFrom->zUuid, pFileTo->zUuid)==0 ){ /* No changes */ - iFrom++; - iTo++; + pFileFrom = manifest_file_next(pFrom, 0); + pFileTo = manifest_file_next(pTo, 0); }else{ - append_file_change_line(mFrom.aFile[iFrom].zName, - mFrom.aFile[iFrom].zUuid, - mTo.aFile[iTo].zUuid, showDetail); - iFrom++; - iTo++; + append_file_change_line(pFileFrom->zName, + pFileFrom->zUuid, + pFileTo->zUuid, showDetail); + pFileFrom = manifest_file_next(pFrom, 0); + pFileTo = manifest_file_next(pTo, 0); } } - manifest_clear(&mFrom); - manifest_clear(&mTo); + manifest_destroy(pFrom); + manifest_destroy(pTo); style_footer(); } /* @@ -761,11 +787,11 @@ } } db_finalize(&q); if( nWiki==0 ){ db_prepare(&q, - "SELECT datetime(mtime), user, comment, type, uuid" + "SELECT datetime(mtime), user, comment, type, uuid, tagid" " FROM event, blob" " WHERE event.objid=%d" " AND blob.rid=%d", rid, rid ); @@ -782,14 +808,19 @@ @ Wiki edit }else if( zType[0]=='t' ){ @ Ticket change }else if( zType[0]=='c' ){ @ Manifest of check-in + }else if( zType[0]=='e' ){ + @ Instance of event + hyperlink_to_event_tagid(db_column_int(&q, 5)); }else{ @ Control file referencing } - hyperlink_to_uuid(zUuid); + if( zType[0]!='e' ){ + hyperlink_to_uuid(zUuid); + } @ - %w(zCom) by hyperlink_to_user(zUser,zDate," on"); hyperlink_to_date(zDate, "."); if( pDownloadName && blob_size(pDownloadName)==0 ){ blob_appendf(pDownloadName, "%.10s.txt", zUuid); @@ -864,23 +895,23 @@ v1 = name_to_rid_www("v1"); v2 = name_to_rid_www("v2"); if( v1==0 || v2==0 ) fossil_redirect_home(); style_header("Diff"); @

- @

+ @
object_description(v1, 1, 0); - @
+ @

- @

+ @
object_description(v2, 1, 0); - @
- @
+ @

+ @

   content_get(v1, &c1);
   content_get(v2, &c2);
   blob_zero(&diff);
-  text_diff(&c1, &c2, &diff, 4);
+  text_diff(&c1, &c2, &diff, 4, 1);
   blob_reset(&c1);
   blob_reset(&c2);
   @ %h(blob_str(&diff))
   @

blob_reset(&diff); @@ -978,27 +1009,27 @@ if( !g.okRead ){ login_needed(); return; } if( rid==0 ) fossil_redirect_home(); if( g.okAdmin ){ const char *zUuid = db_text("", "SELECT uuid FROM blob WHERE rid=%d", rid); if( db_exists("SELECT 1 FROM shun WHERE uuid='%s'", zUuid) ){ - style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", + style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", g.zTop, zUuid); }else{ style_submenu_element("Shun","Shun", "%s/shun?shun=%s#addshun", g.zTop, zUuid); } } style_header("Hex Artifact Content"); zUuid = db_text("?","SELECT uuid FROM blob WHERE rid=%d", rid); @

- @

+ @
blob_zero(&downloadName); object_description(rid, 0, &downloadName); style_submenu_element("Download", "Download", "%s/raw/%T?name=%s", g.zTop, blob_str(&downloadName), zUuid); - @
- @
+ @

+ @

content_get(rid, &content); @

   hexdump(&content);
   @

style_footer(); @@ -1010,25 +1041,26 @@ */ int artifact_from_ci_and_filename(void){ const char *zFilename; const char *zCI; int cirid; - Blob content; - Manifest m; - int i; + Manifest *pManifest; + ManifestFile *pFile; zCI = P("ci"); if( zCI==0 ) return 0; zFilename = P("filename"); if( zFilename==0 ) return 0; cirid = name_to_rid_www("ci"); - if( !content_get(cirid, &content) ) return 0; - if( !manifest_parse(&m, &content) ) return 0; - if( m.type!=CFTYPE_MANIFEST ) return 0; - for(i=0; izName)==0 ){ + int rid = db_int(0, "SELECT rid FROM blob WHERE uuid=%Q", pFile->zUuid); + manifest_destroy(pManifest); + return rid; } } return 0; } @@ -1060,21 +1092,21 @@ if( !g.okRead ){ login_needed(); return; } if( rid==0 ) fossil_redirect_home(); if( g.okAdmin ){ const char *zUuid = db_text("", "SELECT uuid FROM blob WHERE rid=%d", rid); if( db_exists("SELECT 1 FROM shun WHERE uuid='%s'", zUuid) ){ - style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", + style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", g.zTop, zUuid); }else{ style_submenu_element("Shun","Shun", "%s/shun?shun=%s#addshun", g.zTop, zUuid); } } style_header("Artifact Content"); zUuid = db_text("?", "SELECT uuid FROM blob WHERE rid=%d", rid); @

- @

+ @
blob_zero(&downloadName); object_description(rid, 0, &downloadName); style_submenu_element("Download", "Download", "%s/raw/%T?name=%s", g.zTop, blob_str(&downloadName), zUuid); zMime = mimetype_from_name(blob_str(&downloadName)); @@ -1084,25 +1116,25 @@ style_submenu_element("Html", "Html", "%s/artifact?name=%s", g.zTop, zUuid); }else{ renderAsHtml = 1; style_submenu_element("Text", "Text", - "%s/artifact?name=%s&txt=1", g.zTop, zUuid); + "%s/artifact?name=%s&txt=1", g.zTop, zUuid); } }else if( strcmp(zMime, "application/x-fossil-wiki")==0 ){ if( P("txt") ){ style_submenu_element("Wiki", "Wiki", "%s/artifact?name=%s", g.zTop, zUuid); }else{ renderAsWiki = 1; style_submenu_element("Text", "Text", - "%s/artifact?name=%s&txt=1", g.zTop, zUuid); + "%s/artifact?name=%s&txt=1", g.zTop, zUuid); } } } - @
- @
+ @

+ @

content_get(rid, &content); if( renderAsWiki ){ wiki_convert(&content, 0, 0); }else if( renderAsHtml ){ @

@@ -1115,11 +1147,11 @@ @

       @ %h(blob_str(&content))
       @

style_submenu_element("Hex","Hex", "%s/hexdump?name=%s", g.zTop, zUuid); }else if( strncmp(zMime, "image/", 6)==0 ){ - @

+ @

style_submenu_element("Hex","Hex", "%s/hexdump?name=%s", g.zTop, zUuid); }else{ @

       hexdump(&content);
       @

@@ -1135,59 +1167,56 @@ ** ** Show the details of a ticket change control artifact. */ void tinfo_page(void){ int rid; - Blob content; char *zDate; const char *zUuid; char zTktName[20]; - Manifest m; + Manifest *pTktChng; login_check_credentials(); if( !g.okRdTkt ){ login_needed(); return; } rid = name_to_rid_www("name"); if( rid==0 ){ fossil_redirect_home(); } zUuid = db_text("", "SELECT uuid FROM blob WHERE rid=%d", rid); if( g.okAdmin ){ if( db_exists("SELECT 1 FROM shun WHERE uuid='%s'", zUuid) ){ - style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", + style_submenu_element("Unshun","Unshun", "%s/shun?uuid=%s&sub=1", g.zTop, zUuid); }else{ style_submenu_element("Shun","Shun", "%s/shun?shun=%s#addshun", g.zTop, zUuid); } } - content_get(rid, &content); - if( manifest_parse(&m, &content)==0 ){ - fossil_redirect_home(); - } - if( m.type!=CFTYPE_TICKET ){ + pTktChng = manifest_get(rid, CFTYPE_TICKET); + if( pTktChng==0 ){ fossil_redirect_home(); } style_header("Ticket Change Details"); - zDate = db_text(0, "SELECT datetime(%.12f)", m.rDate); - memcpy(zTktName, m.zTicketUuid, 10); + zDate = db_text(0, "SELECT datetime(%.12f)", pTktChng->rDate); + memcpy(zTktName, pTktChng->zTicketUuid, 10); zTktName[10] = 0; if( g.okHistory ){ - @

+ @

@ - @

By %h(m.zUser) on %s(zDate). See also: + @

By %h(pTktChng->zUser) on %s(zDate). See also: @ artifact content, and - @ ticket history - @

+ @ ticket + @ history

}else{ @

@ - @

By %h(m.zUser) on %s(zDate). + @

By %h(pTktChng->zUser) on %s(zDate). @

} @ @

+ }else{ + @

+ if( fPropagate ){ + @ + }else{ + @ + } + @ Propagate color to descendants
+ } + if( strcmp(zDefaultColor, aColor[i].zColor)==0 ){ + @ + stdClrFound=1; + }else{ + @ + } + @ %h(aColor[i].zCName)
+ @ + }else{ + @						+ @ + } + @ %h(aColor[i].zCName) + @ + @

Preview:

@ if( zNewColor && zNewColor[0] ){ - @
+ @
}else{ @
} wiki_convert(&comment, 0, WIKI_INLINE); blob_zero(&suffix); @@ -1451,71 +1544,45 @@ db_finalize(&q); blob_appendf(&suffix, ")"); @ %s(blob_str(&suffix)) @
@

Make changes to attributes of check-in @ [%s(zUuid)]:

+ @

login_insert_csrf_secret(); - @ + @ @ @ @ @ @ @ @ @ @ @ @ @ @ if( is_a_leaf(rid) && !db_exists("SELECT 1 FROM tagxref " " WHERE tagid=%d AND rid=%d AND tagtype>0", TAG_CLOSED, rid) ){ @ @ } @ @

User:

- @ + @ @

Comment:

@ @

Check-in Time:

- @ + @ @

Background Color:

- @ - @ - @ - for(i=0; i - }else{ - @ - if( (i%6)==5 && i+1 - } - } - @ - @

- if( fPropagateColor ){ - @ - }else{ - @ - } - @ Propagate color to descendants
- } - if( strcmp(zNewColor, aColor[i].zColor)==0 ){ - @ - }else{ - @ - } - @ %h(aColor[i].zCName)

+ render_color_chooser(fPropagateColor, zNewColor, "pclr", "clr", "clrcust"); @

Tags:

- @ + @ @ Add the following new tag name to this check-in: - @ + @ db_prepare(&q, "SELECT tag.tagid, tagname FROM tagxref, tag" " WHERE tagxref.rid=%d AND tagtype>0 AND tagxref.tagid=tag.tagid" " ORDER BY CASE WHEN tagname GLOB 'sym-*' THEN substr(tagname,5)" " ELSE tagname END", @@ -1525,13 +1592,13 @@ int tagid = db_column_int(&q, 0); const char *zTagName = db_column_text(&q, 1); char zLabel[30]; sprintf(zLabel, "c%d", tagid); if( P(zLabel) ){ - @
+ @
}else{ - @
+ @
} if( strncmp(zTagName, "sym-", 4)==0 ){ @ Cancel tag %h(&zTagName[4]) }else{ @ Cancel special tag %h(zTagName) @@ -1540,33 +1607,33 @@ db_finalize(&q); @

Branching:

- @ + @ @ Make this check-in the start of a new branch named: - @ + @ @

Leaf Closure:

- @ + @ @ Mark this leaf as "closed" so that it no longer appears on the @ "leaves" page and is no longer labeled as a "Leaf". @

- @ - @ - @ + @ + @ + @ @

- @ + @

/* for Sleep */ -# define sleep Sleep /* windows does not have sleep, but Sleep */ +# if defined(__MINGW32__) || defined(_MSC_VER) +# define sleep Sleep /* windows does not have sleep, but Sleep */ +# endif #endif #include /* ** Return the name of the login cookie */ static char *login_cookie_name(void){ static char *zCookieName = 0; if( zCookieName==0 ){ - int n = strlen(g.zTop); - zCookieName = malloc( n*2+16 ); - /* 0123456789 12345 */ - strcpy(zCookieName, "fossil_login_"); - encode16((unsigned char*)g.zTop, (unsigned char*)&zCookieName[13], n); + unsigned int h = 0; + const char *z = g.zBaseURL; + while( *z ){ h = (h<<3) ^ (h>>26) ^ *(z++); } + zCookieName = mprintf("fossil_login_%08x", h); } return zCookieName; } /* @@ -151,21 +152,21 @@ if( db_int(1, "SELECT 0 FROM user" " WHERE uid=%d AND (pw=%Q OR pw=%Q)", g.userUid, zPasswd, zSha1Pw) ){ sleep(1); zErrMsg = - @

+ @

@ You entered an incorrect old password while attempting to change @ your password. Your password is unchanged. - @

+ @

; }else if( strcmp(zNew1,zNew2)!=0 ){ zErrMsg = - @

+ @

@ The two copies of your new passwords do not match. @ Your password is unchanged. - @

+ @

; }else{ char *zNewPw = sha1_shared_secret(zNew1, g.zLogin); db_multi_exec( "UPDATE user SET pw=%Q WHERE uid=%d", zNewPw, g.userUid @@ -204,13 +205,13 @@ zUsername, zPasswd, zSha1Pw ); if( uid<=0 ){ sleep(1); zErrMsg = - @

+ @

@ You entered an unknown user or an incorrect password. - @

+ @

; }else{ char *zCookie; const char *zCookieName = login_cookie_name(); const char *zExpire = db_get("cookie-expire","8766"); @@ -227,38 +228,38 @@ redirect_to_g(); } } style_header("Login/Logout"); @ %s(zErrMsg) - @

+ @ if( P("g") ){ - @ + @ } - @ + @

@ - @ + @ if( anonFlag ){ - @ + @ }else{ - @ + @ } @ @ - @ - @ + @ + @ @ if( g.zLogin==0 ){ zAnonPw = db_text(0, "SELECT pw FROM user" " WHERE login='anonymous'" " AND cap!=''"); } @ @ - @ + @ @ @

User ID:	User ID:
Password:		Password:

- @ + @ if( g.zLogin==0 ){ @

Enter }else{ @

You are currently logged in as %h(g.zLogin)

To change your login to a different user, enter @@ -271,46 +272,46 @@ unsigned int uSeed = captcha_seed(); char const *zDecoded = captcha_decode(uSeed); int bAutoCaptcha = db_get_boolean("auto-captcha", 1); char *zCaptcha = captcha_render(zDecoded); - @ - @

Visitors may enter anonymous as the user-ID with + @

+ @ Visitors may enter anonymous as the user-ID with @ the 8-character hexadecimal password shown below:

- @ "); + for(i=0; iout,"\n"); + } + fprintf(p->out,"\n"); + } + if( azArg==0 ) break; + fprintf(p->out,""); + for(i=0; iout,"\n"); + } + fprintf(p->out,"\n"); + break; + } + case MODE_Tcl: { + if( p->cnt++==0 && p->showHeader ){ + for(i=0; iout,azCol[i] ? azCol[i] : ""); + fprintf(p->out, "%s", p->separator); + } + fprintf(p->out,"\n"); + } + if( azArg==0 ) break; + for(i=0; iout, azArg[i] ? azArg[i] : p->nullvalue); + fprintf(p->out, "%s", p->separator); + } + fprintf(p->out,"\n"); + break; + } + case MODE_Csv: { + if( p->cnt++==0 && p->showHeader ){ + for(i=0; iout,"\n"); + } + if( azArg==0 ) break; + for(i=0; iout,"\n"); + break; + } + case MODE_Insert: { + p->cnt++; + if( azArg==0 ) break; + fprintf(p->out,"INSERT INTO %s VALUES(",p->zDestTable); + for(i=0; i0 ? ",": ""; + if( (azArg[i]==0) || (aiType && aiType[i]==SQLITE_NULL) ){ + fprintf(p->out,"%sNULL",zSep); + }else if( aiType && aiType[i]==SQLITE_TEXT ){ + if( zSep[0] ) fprintf(p->out,"%s",zSep); + output_quoted_string(p->out, azArg[i]); + }else if( aiType && (aiType[i]==SQLITE_INTEGER || aiType[i]==SQLITE_FLOAT) ){ + fprintf(p->out,"%s%s",zSep, azArg[i]); + }else if( aiType && aiType[i]==SQLITE_BLOB && p->pStmt ){ + const void *pBlob = sqlite3_column_blob(p->pStmt, i); + int nBlob = sqlite3_column_bytes(p->pStmt, i); + if( zSep[0] ) fprintf(p->out,"%s",zSep); + output_hex_blob(p->out, pBlob, nBlob); + }else if( isNumber(azArg[i], 0) ){ + fprintf(p->out,"%s%s",zSep, azArg[i]); + }else{ + if( zSep[0] ) fprintf(p->out,"%s",zSep); + output_quoted_string(p->out, azArg[i]); + } + } + fprintf(p->out,");\n"); + break; + } + } + return 0; +} + +/* +** This is the callback routine that the SQLite library +** invokes for each row of a query result. +*/ +static int callback(void *pArg, int nArg, char **azArg, char **azCol){ + /* since we don't have type info, call the shell_callback with a NULL value */ + return shell_callback(pArg, nArg, azArg, azCol, NULL); +} + +/* +** Set the destination table field of the callback_data structure to +** the name of the table given. Escape any quote characters in the +** table name. +*/ +static void set_table_name(struct callback_data *p, const char *zName){ + int i, n; + int needQuote; + char *z; + + if( p->zDestTable ){ + free(p->zDestTable); + p->zDestTable = 0; + } + if( zName==0 ) return; + needQuote = !isalpha((unsigned char)*zName) && *zName!='_'; + for(i=n=0; zName[i]; i++, n++){ + if( !isalnum((unsigned char)zName[i]) && zName[i]!='_' ){ + needQuote = 1; + if( zName[i]=='\'' ) n++; + } + } + if( needQuote ) n += 2; + z = p->zDestTable = malloc( n+1 ); + if( z==0 ){ + fprintf(stderr,"Error: out of memory\n"); + exit(1); + } + n = 0; + if( needQuote ) z[n++] = '\''; + for(i=0; zName[i]; i++){ + z[n++] = zName[i]; + if( zName[i]=='\'' ) z[n++] = '\''; + } + if( needQuote ) z[n++] = '\''; + z[n] = 0; +} + +/* zIn is either a pointer to a NULL-terminated string in memory obtained +** from malloc(), or a NULL pointer. The string pointed to by zAppend is +** added to zIn, and the result returned in memory obtained from malloc(). +** zIn, if it was not NULL, is freed. +** +** If the third argument, quote, is not '\0', then it is used as a +** quote character for zAppend. +*/ +static char *appendText(char *zIn, char const *zAppend, char quote){ + int len; + int i; + int nAppend = strlen30(zAppend); + int nIn = (zIn?strlen30(zIn):0); + + len = nAppend+nIn+1; + if( quote ){ + len += 2; + for(i=0; iout ){ + + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_MEMORY_USED, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Memory Used: %d (max %d) bytes\n", iCur, iHiwtr); + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_MALLOC_COUNT, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Number of Allocations: %d (max %d)\n", iCur, iHiwtr); +/* +** Not currently used by the CLI. +** iHiwtr = iCur = -1; +** sqlite3_status(SQLITE_STATUS_PAGECACHE_USED, &iCur, &iHiwtr, bReset); +** fprintf(pArg->out, "Number of Pcache Pages Used: %d (max %d) pages\n", iCur, iHiwtr); +*/ + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_PAGECACHE_OVERFLOW, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Number of Pcache Overflow Bytes: %d (max %d) bytes\n", iCur, iHiwtr); +/* +** Not currently used by the CLI. +** iHiwtr = iCur = -1; +** sqlite3_status(SQLITE_STATUS_SCRATCH_USED, &iCur, &iHiwtr, bReset); +** fprintf(pArg->out, "Number of Scratch Allocations Used: %d (max %d)\n", iCur, iHiwtr); +*/ + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_SCRATCH_OVERFLOW, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Number of Scratch Overflow Bytes: %d (max %d) bytes\n", iCur, iHiwtr); + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_MALLOC_SIZE, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Largest Allocation: %d bytes\n", iHiwtr); + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_PAGECACHE_SIZE, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Largest Pcache Allocation: %d bytes\n", iHiwtr); + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_SCRATCH_SIZE, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Largest Scratch Allocation: %d bytes\n", iHiwtr); +#ifdef YYTRACKMAXSTACKDEPTH + iHiwtr = iCur = -1; + sqlite3_status(SQLITE_STATUS_PARSER_STACK, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Deepest Parser Stack: %d (max %d)\n", iCur, iHiwtr); +#endif + } + + if( pArg && pArg->out && db ){ + iHiwtr = iCur = -1; + sqlite3_db_status(db, SQLITE_DBSTATUS_LOOKASIDE_USED, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Lookaside Slots Used: %d (max %d)\n", iCur, iHiwtr); + iHiwtr = iCur = -1; + sqlite3_db_status(db, SQLITE_DBSTATUS_CACHE_USED, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Pager Heap Usage: %d bytes\n", iCur); + iHiwtr = iCur = -1; + sqlite3_db_status(db, SQLITE_DBSTATUS_SCHEMA_USED, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Schema Heap Usage: %d bytes\n", iCur); + iHiwtr = iCur = -1; + sqlite3_db_status(db, SQLITE_DBSTATUS_STMT_USED, &iCur, &iHiwtr, bReset); + fprintf(pArg->out, "Statement Heap/Lookaside Usage: %d bytes\n", iCur); + } + + if( pArg && pArg->out && db && pArg->pStmt ){ + iCur = sqlite3_stmt_status(pArg->pStmt, SQLITE_STMTSTATUS_FULLSCAN_STEP, bReset); + fprintf(pArg->out, "Fullscan Steps: %d\n", iCur); + iCur = sqlite3_stmt_status(pArg->pStmt, SQLITE_STMTSTATUS_SORT, bReset); + fprintf(pArg->out, "Sort Operations: %d\n", iCur); + iCur = sqlite3_stmt_status(pArg->pStmt, SQLITE_STMTSTATUS_AUTOINDEX, bReset); + fprintf(pArg->out, "Autoindex Inserts: %d\n", iCur); + } + + return 0; +} + +/* +** Execute a statement or set of statements. Print +** any result rows/columns depending on the current mode +** set via the supplied callback. +** +** This is very similar to SQLite's built-in sqlite3_exec() +** function except it takes a slightly different callback +** and callback data argument. +*/ +static int shell_exec( + sqlite3 *db, /* An open database */ + const char *zSql, /* SQL to be evaluated */ + int (*xCallback)(void*,int,char**,char**,int*), /* Callback function */ + /* (not the same as sqlite3_exec) */ + struct callback_data *pArg, /* Pointer to struct callback_data */ + char **pzErrMsg /* Error msg written here */ +){ + sqlite3_stmt *pStmt = NULL; /* Statement to execute. */ + int rc = SQLITE_OK; /* Return Code */ + const char *zLeftover; /* Tail of unprocessed SQL */ + + if( pzErrMsg ){ + *pzErrMsg = NULL; + } + + while( zSql[0] && (SQLITE_OK == rc) ){ + rc = sqlite3_prepare_v2(db, zSql, -1, &pStmt, &zLeftover); + if( SQLITE_OK != rc ){ + if( pzErrMsg ){ + *pzErrMsg = save_err_msg(db); + } + }else{ + if( !pStmt ){ + /* this happens for a comment or white-space */ + zSql = zLeftover; + while( isspace(zSql[0]) ) zSql++; + continue; + } + + /* save off the prepared statment handle and reset row count */ + if( pArg ){ + pArg->pStmt = pStmt; + pArg->cnt = 0; + } + + /* echo the sql statement if echo on */ + if( pArg && pArg->echoOn ){ + const char *zStmtSql = sqlite3_sql(pStmt); + fprintf(pArg->out, "%s\n", zStmtSql ? zStmtSql : zSql); + } + + /* perform the first step. this will tell us if we + ** have a result set or not and how wide it is. + */ + rc = sqlite3_step(pStmt); + /* if we have a result set... */ + if( SQLITE_ROW == rc ){ + /* if we have a callback... */ + if( xCallback ){ + /* allocate space for col name ptr, value ptr, and type */ + int nCol = sqlite3_column_count(pStmt); + void *pData = sqlite3_malloc(3*nCol*sizeof(const char*) + 1); + if( !pData ){ + rc = SQLITE_NOMEM; + }else{ + char **azCols = (char **)pData; /* Names of result columns */ + char **azVals = &azCols[nCol]; /* Results */ + int *aiTypes = (int *)&azVals[nCol]; /* Result types */ + int i; + assert(sizeof(int) <= sizeof(char *)); + /* save off ptrs to column names */ + for(i=0; istatsOn ){ + display_stats(db, pArg, 0); + } + + /* Finalize the statement just executed. If this fails, save a + ** copy of the error message. Otherwise, set zSql to point to the + ** next statement to execute. */ + rc = sqlite3_finalize(pStmt); + if( rc==SQLITE_OK ){ + zSql = zLeftover; + while( isspace(zSql[0]) ) zSql++; + }else if( pzErrMsg ){ + *pzErrMsg = save_err_msg(db); + } + + /* clear saved stmt handle */ + if( pArg ){ + pArg->pStmt = NULL; + } + } + } /* end while */ + + return rc; +} + + +/* +** This is a different callback routine used for dumping the database. +** Each row received by this callback consists of a table name, +** the table type ("index" or "table") and SQL to create the table. +** This routine should print text sufficient to recreate the table. +*/ +static int dump_callback(void *pArg, int nArg, char **azArg, char **azCol){ + int rc; + const char *zTable; + const char *zType; + const char *zSql; + const char *zPrepStmt = 0; + struct callback_data *p = (struct callback_data *)pArg; + + UNUSED_PARAMETER(azCol); + if( nArg!=3 ) return 1; + zTable = azArg[0]; + zType = azArg[1]; + zSql = azArg[2]; + + if( strcmp(zTable, "sqlite_sequence")==0 ){ + zPrepStmt = "DELETE FROM sqlite_sequence;\n"; + }else if( strcmp(zTable, "sqlite_stat1")==0 ){ + fprintf(p->out, "ANALYZE sqlite_master;\n"); + }else if( strncmp(zTable, "sqlite_", 7)==0 ){ + return 0; + }else if( strncmp(zSql, "CREATE VIRTUAL TABLE", 20)==0 ){ + char *zIns; + if( !p->writableSchema ){ + fprintf(p->out, "PRAGMA writable_schema=ON;\n"); + p->writableSchema = 1; + } + zIns = sqlite3_mprintf( + "INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)" + "VALUES('table','%q','%q',0,'%q');", + zTable, zTable, zSql); + fprintf(p->out, "%s\n", zIns); + sqlite3_free(zIns); + return 0; + }else{ + fprintf(p->out, "%s;\n", zSql); + } + + if( strcmp(zType, "table")==0 ){ + sqlite3_stmt *pTableInfo = 0; + char *zSelect = 0; + char *zTableInfo = 0; + char *zTmp = 0; + int nRow = 0; + + zTableInfo = appendText(zTableInfo, "PRAGMA table_info(", 0); + zTableInfo = appendText(zTableInfo, zTable, '"'); + zTableInfo = appendText(zTableInfo, ");", 0); + + rc = sqlite3_prepare(p->db, zTableInfo, -1, &pTableInfo, 0); + free(zTableInfo); + if( rc!=SQLITE_OK || !pTableInfo ){ + return 1; + } + + zSelect = appendText(zSelect, "SELECT 'INSERT INTO ' || ", 0); + zTmp = appendText(zTmp, zTable, '"'); + if( zTmp ){ + zSelect = appendText(zSelect, zTmp, '\''); + } + zSelect = appendText(zSelect, " || ' VALUES(' || ", 0); + rc = sqlite3_step(pTableInfo); + while( rc==SQLITE_ROW ){ + const char *zText = (const char *)sqlite3_column_text(pTableInfo, 1); + zSelect = appendText(zSelect, "quote(", 0); + zSelect = appendText(zSelect, zText, '"'); + rc = sqlite3_step(pTableInfo); + if( rc==SQLITE_ROW ){ + zSelect = appendText(zSelect, ") || ',' || ", 0); + }else{ + zSelect = appendText(zSelect, ") ", 0); + } + nRow++; + } + rc = sqlite3_finalize(pTableInfo); + if( rc!=SQLITE_OK || nRow==0 ){ + free(zSelect); + return 1; + } + zSelect = appendText(zSelect, "|| ')' FROM ", 0); + zSelect = appendText(zSelect, zTable, '"'); + + rc = run_table_dump_query(p->out, p->db, zSelect, zPrepStmt); + if( rc==SQLITE_CORRUPT ){ + zSelect = appendText(zSelect, " ORDER BY rowid DESC", 0); + rc = run_table_dump_query(p->out, p->db, zSelect, 0); + } + if( zSelect ) free(zSelect); + } + return 0; +} + +/* +** Run zQuery. Use dump_callback() as the callback routine so that +** the contents of the query are output as SQL statements. +** +** If we get a SQLITE_CORRUPT error, rerun the query after appending +** "ORDER BY rowid DESC" to the end. +*/ +static int run_schema_dump_query( + struct callback_data *p, + const char *zQuery, + char **pzErrMsg +){ + int rc; + rc = sqlite3_exec(p->db, zQuery, dump_callback, p, pzErrMsg); + if( rc==SQLITE_CORRUPT ){ + char *zQ2; + int len = strlen30(zQuery); + if( pzErrMsg ) sqlite3_free(*pzErrMsg); + zQ2 = malloc( len+100 ); + if( zQ2==0 ) return rc; + sqlite3_snprintf(sizeof(zQ2), zQ2, "%s ORDER BY rowid DESC", zQuery); + rc = sqlite3_exec(p->db, zQ2, dump_callback, p, pzErrMsg); + free(zQ2); + } + return rc; +} + +/* +** Text of a help message +*/ +static char zHelp[] = + ".backup ?DB? FILE Backup DB (default \"main\") to FILE\n" + ".bail ON|OFF Stop after hitting an error. Default OFF\n" + ".databases List names and files of attached databases\n" + ".dump ?TABLE? ... Dump the database in an SQL text format\n" + " If TABLE specified, only dump tables matching\n" + " LIKE pattern TABLE.\n" + ".echo ON|OFF Turn command echo on or off\n" + ".exit Exit this program\n" + ".explain ?ON|OFF? Turn output mode suitable for EXPLAIN on or off.\n" + " With no args, it turns EXPLAIN on.\n" + ".header(s) ON|OFF Turn display of headers on or off\n" + ".help Show this message\n" + ".import FILE TABLE Import data from FILE into TABLE\n" + ".indices ?TABLE? Show names of all indices\n" + " If TABLE specified, only show indices for tables\n" + " matching LIKE pattern TABLE.\n" +#ifdef SQLITE_ENABLE_IOTRACE + ".iotrace FILE Enable I/O diagnostic logging to FILE\n" +#endif +#ifndef SQLITE_OMIT_LOAD_EXTENSION + ".load FILE ?ENTRY? Load an extension library\n" +#endif + ".log FILE|off Turn logging on or off. FILE can be stderr/stdout\n" + ".mode MODE ?TABLE? Set output mode where MODE is one of:\n" + " csv Comma-separated values\n" + " column Left-aligned columns. (See .width)\n" + " html HTML

+    @      @ %s(zCaptcha)
     @ 
     if( bAutoCaptcha ) {
         @ 
+        @           document.getElementById('p').value='%s(zDecoded)';" />
     }
-    @ 
+    @ 
     free(zCaptcha);
   }
   if( g.zLogin ){
-    @ 

+    @ 
     @ To log off the system (and delete your login cookie)
-    @  press the following button:

-    @ 
+    @  press the following button:

+    @ 
   }
   @ 
   if( g.okPassword ){
-    @ 

+    @ 
     @ To change your password, enter your old password and your
     @ new password twice below then press the "Change Password"
     @ button.
-    @ 
+    @ 
     @ 
-    @ 
-    @ 
-    @ 
-    @ 
-    @ 
-    @ 
+    @ 
+    @ 
+    @ 
+    @ 
+    @ 
+    @ 
     @ 
-    @ 
+    @ 
     @ Old Password: 
New Password: 
Repeat New Password: 
Old Password: 
New Password: 
Repeat New Password: 

     @ 
   }
   style_footer();
 }
@@ -351,11 +352,11 @@
   }
 
   /* Check the login cookie to see if it matches a known valid user.
   */
   if( uid==0 && (zCookie = P(login_cookie_name()))!=0 ){
-    if( isdigit(zCookie[0]) ){
+    if( fossil_isdigit(zCookie[0]) ){
       /* Cookies of the form "uid/randomness".  There must be a
       ** corresponding entry in the user table. */
       uid = db_int(0, 
             "SELECT uid FROM user"
             " WHERE uid=%d"
@@ -594,22 +595,22 @@
   if( !g.okHistory &&
       db_exists("SELECT 1 FROM user"
                 " WHERE login='anonymous'"
                 "   AND cap LIKE '%%h%%'") ){
     const char *zUrl = PD("REQUEST_URI", "index");
-    @ Many hyperlinks are disabled.

-    @ Use anonymous login
+    @ 
Many hyperlinks are disabled.

+    @ Use anonymous login
     @ to enable hyperlinks.
   }
 }
 
 /*
 ** While rendering a form, call this routine to add the Anti-CSRF token
 ** as a hidden element of the form.
 */
 void login_insert_csrf_secret(void){
-  @ 
+  @ 
 }
 
 /*
 ** Before using the results of a form, first call this routine to verify
 ** that ths Anti-CSRF token is present and is valid.  If the Anti-CSRF token

Index: src/main.c
==================================================================
--- src/main.c
+++ src/main.c
@@ -83,13 +83,15 @@
   FILE *httpOut;          /* Send HTTP output here */
   int xlinkClusterOnly;   /* Set when cloning.  Only process clusters */
   int fTimeFormat;        /* 1 for UTC.  2 for localtime.  0 not yet selected */
   int *aCommitFile;       /* Array of files to be committed */
   int markPrivate;        /* All new artifacts are private if true */
+  int clockSkewSeen;      /* True if clocks on client and server out of sync */
 
   int urlIsFile;          /* True if a "file:" url */
   int urlIsHttps;         /* True if a "https:" url */
+  int urlIsSsh;           /* True if an "ssh:" url */
   char *urlName;          /* Hostname for http: or filename for file: */
   char *urlHostname;      /* The HOST: parameter on http headers */
   char *urlProtocol;      /* "http" or "https" */
   int urlPort;            /* TCP port number for http: or https: */
   int urlDfltPort;        /* The default port for the given protocol */
@@ -96,10 +98,11 @@
   char *urlPath;          /* Pathname for http: */
   char *urlUser;          /* User id for http: */
   char *urlPasswd;        /* Password for http: */
   char *urlCanonical;     /* Canonical representation of the URL */
   char *urlProxyAuth;     /* Proxy-Authorizer: string */
+  char *urlFossil;        /* The path of the ?fossil=path suffix on ssh: */
   int dontKeepUrl;        /* Do not persist the URL */
 
   const char *zLogin;     /* Login name.  "" if not logged in. */
   int noPswd;             /* Logged in without password (on 127.0.0.1) */
   int userUid;            /* Integer user id */
@@ -220,32 +223,39 @@
 */
 int main(int argc, char **argv){
   const char *zCmdName = "unknown";
   int idx;
   int rc;
+  int mightBeCgi;
 
   sqlite3_config(SQLITE_CONFIG_LOG, fossil_sqlite_log, 0);
   g.now = time(0);
   g.argc = argc;
   g.argv = argv;
-  if( getenv("GATEWAY_INTERFACE")!=0 ){
-    zCmdName = "cgi";
-  }else if( argc<2 ){
-    fprintf(stderr, "Usage: %s COMMAND ...\n"
-                    "\"%s help\" for a list of available commands\n"
-                    "\"%s help COMMAND\" for specific details\n",
-                    argv[0], argv[0], argv[0]);
-    fossil_exit(1);
+  mightBeCgi = getenv("GATEWAY_INTERFACE")!=0;
+  if( argc<2 ){
+    if( mightBeCgi ){
+      zCmdName = "cgi";
+    }else{
+      fprintf(stderr, "Usage: %s COMMAND ...\n"
+                      "\"%s help\" for a list of available commands\n"
+                      "\"%s help COMMAND\" for specific details\n",
+                      argv[0], argv[0], argv[0]);
+      fossil_exit(1);
+    }
   }else{
     g.fQuiet = find_option("quiet", 0, 0)!=0;
     g.fSqlTrace = find_option("sqltrace", 0, 0)!=0;
     g.fSqlPrint = find_option("sqlprint", 0, 0)!=0;
     g.fHttpTrace = find_option("httptrace", 0, 0)!=0;
     g.zLogin = find_option("user", "U", 1);
     zCmdName = argv[1];
   }
   rc = name_search(zCmdName, aCommand, count(aCommand), &idx);
+  if( rc==1 && mightBeCgi ){
+    rc = name_search("cgi", aCommand, count(aCommand), &idx);
+  }
   if( rc==1 ){
     fprintf(stderr,"%s: unknown command: %s\n"
                    "%s: use \"help\" for more information\n",
                    argv[0], zCmdName, argv[0]);
     fossil_exit(1);
@@ -288,11 +298,11 @@
   va_start(ap, zFormat);
   z = vmprintf(zFormat, ap);
   va_end(ap);
   if( g.cgiOutput && once ){
     once = 0;
-    cgi_printf("%h", z);
+    cgi_printf("%h", z);
     cgi_reply();
   }else{
     fprintf(stderr, "%s: %s\n", g.argv[0], z);
   }
   db_force_rollback();
@@ -305,11 +315,11 @@
   va_start(ap, zFormat);
   z = vmprintf(zFormat, ap);
   va_end(ap);
   if( g.cgiOutput ){
     g.cgiOutput = 0;
-    cgi_printf("%h", z);
+    cgi_printf("%h", z);
     cgi_reply();
   }else{
     fprintf(stderr, "%s: %s\n", g.argv[0], z);
   }
   db_force_rollback();
@@ -333,11 +343,11 @@
   va_start(ap, zFormat);
   z = vmprintf(zFormat, ap);
   va_end(ap);
   if( g.cgiOutput ){
     g.cgiOutput = 0;
-    cgi_printf("%h", z);
+    cgi_printf("%h", z);
     cgi_reply();
   }else{
     fprintf(stderr, "%s: %s\n", g.argv[0], z);
   }
   db_force_rollback();
@@ -351,15 +361,54 @@
   va_list ap;
   va_start(ap, zFormat);
   z = vmprintf(zFormat, ap);
   va_end(ap);
   if( g.cgiOutput ){
-    cgi_printf("%h", z);
+    cgi_printf("%h", z);
   }else{
     fprintf(stderr, "%s: %s\n", g.argv[0], z);
   }
 }
+
+/*
+** Malloc and free routines that cannot fail
+*/
+void *fossil_malloc(size_t n){
+  void *p = malloc(n);
+  if( p==0 ) fossil_panic("out of memory");
+  return p;
+}
+void fossil_free(void *p){
+  free(p);
+}
+void *fossil_realloc(void *p, size_t n){
+  p = realloc(p, n);
+  if( p==0 ) fossil_panic("out of memory");
+  return p;
+}
+
+/*
+** This function implements a cross-platform "system()" interface.
+*/
+int fossil_system(const char *zOrigCmd){
+  int rc;
+#if defined(_WIN32)
+  /* On windows, we have to put double-quotes around the entire command.
+  ** Who knows why - this is just the way windows works.
+  */
+  char *zNewCmd = mprintf("\"%s\"", zOrigCmd);
+  rc = system(zNewCmd);
+  free(zNewCmd);
+#else
+  /* On unix, evaluate the command directly.
+  */
+  rc = system(zOrigCmd);
+#endif 
+  return rc; 
+}
+
+
 
 /*
 ** Return a name for an SQLite error code
 */
 static const char *sqlite_error_code_name(int iCode){
@@ -588,10 +637,62 @@
       z++;
     }
   }
   putchar('\n');
 }
+
+/*
+** WEBPAGE: help
+** URL: /help?cmd=CMD
+*/
+void help_page(void){
+    const char * zCmd = P("cmd");
+    
+    style_header("Command line help %s%s",zCmd?" - ":"",zCmd?zCmd:"");
+    if( zCmd ){
+      int rc, idx;
+      char *z, *s, *d;
+
+      @ %s(zCmd)
+      rc = name_search(zCmd, aCommand, count(aCommand), &idx);
+      if( rc==1 ){
+        @ unknown command: %s(zCmd)
+      }else if( rc==2 ){
+        @ ambiguous command prefix: %s(zCmd)
+      }else{
+        z = (char*)aCmdHelp[idx];
+        if( z==0 ){
+          @ no help available for the %s(aCommand[idx].zName) command
+        }else{
+          z=s=d=mprintf("%s",z);
+	  while( *s ){
+	    if( *s=='%' && strncmp(s, "%fossil", 7)==0 ){
+	      s++;
+	    }else{
+	      *d++ = *s++;
+	    }
+	  }
+	  *d = 0;
+	  @ %s(z)
+	  free(z);
+	}
+      }
+      @ 
available commands in fossil
+      @ version %s(MANIFEST_VERSION" "MANIFEST_DATE) UTC
+    }else{
+      int i;
+      
+      @ Available commands
+      for(i=0; i
+        @ %s(aCommand[i].zName)
+      }
+      @ fossil version %s(MANIFEST_VERSION" "MANIFEST_DATE) UTC
+    }
+    style_footer();
+}
 
 /*
 ** Set the g.zBaseURL value to the full URL for the toplevel of
 ** the fossil tree.  Set g.zTop to g.zBaseURL without the
 ** leading "http://" and the host and port.
@@ -630,11 +731,11 @@
 **
 ** Assume the user-id and group-id of the repository, or if zRepo
 ** is a directory, of that directory.
 */
 static char *enter_chroot_jail(char *zRepo){
-#if !defined(__MINGW32__)
+#if !defined(_WIN32)
   if( getuid()==0 ){
     int i;
     struct stat sStat;
     Blob dir;
     char *zDir;
@@ -702,11 +803,11 @@
 
     /* To avoid mischief, make sure the repository basename contains no
     ** characters other than alphanumerics, "-", and "_".
     */
     for(j=strlen(g.zRepositoryName)+1, k=0; kVERSION.h
 	awk '{ printf "#define MANIFEST_VERSION \"[%.10s]\"\n", $$1}'  $(SRCDIR)/../manifest.uuid >>VERSION.h
 	awk '$$1=="D"{printf "#define MANIFEST_DATE \"%s %s\"\n", substr($$2,1,10),substr($$2,12)}'  $(SRCDIR)/../manifest >>VERSION.h
 
-$(APPNAME):	headers $(OBJ) $(OBJDIR)/sqlite3.o $(OBJDIR)/th.o $(OBJDIR)/th_lang.o
-	$(TCC) -o $(APPNAME) $(OBJ) $(OBJDIR)/sqlite3.o $(OBJDIR)/th.o $(OBJDIR)/th_lang.o $(LIB)
+EXTRAOBJ =  $(OBJDIR)/sqlite3.o  $(OBJDIR)/shell.o  $(OBJDIR)/th.o  $(OBJDIR)/th_lang.o
+
+$(APPNAME):	headers $(OBJ) $(EXTRAOBJ)
+	$(TCC) -o $(APPNAME) $(OBJ) $(EXTRAOBJ) $(LIB)
 
 # This rule prevents make from using its default rules to try build
 # an executable named "manifest" out of the file named "manifest.c"
 #
 $(SRCDIR)/../manifest:	
@@ -270,16 +278,16 @@
 	# noop
 
 clean:	
 	rm -f $(OBJDIR)/*.o *_.c $(APPNAME) VERSION.h
 	rm -f translate makeheaders mkindex page_index.h headers
-	rm -f add.h allrepo.h attach.h bag.h blob.h branch.h browse.h captcha.h cgi.h checkin.h checkout.h clearsign.h clone.h comformat.h configure.h content.h db.h delta.h deltacmd.h descendants.h diff.h diffcmd.h doc.h encode.h file.h finfo.h graph.h http.h http_socket.h http_ssl.h http_transport.h info.h login.h main.h manifest.h md5.h merge.h merge3.h name.h pivot.h pqueue.h printf.h rebuild.h report.h rss.h schema.h search.h setup.h sha1.h shun.h skins.h stat.h style.h sync.h tag.h th_main.h timeline.h tkt.h tktsetup.h undo.h update.h url.h user.h verify.h vfile.h wiki.h wikiformat.h winhttp.h xfer.h zip.h
+	rm -f add.h allrepo.h attach.h bag.h blob.h branch.h browse.h captcha.h cgi.h checkin.h checkout.h clearsign.h clone.h comformat.h configure.h content.h db.h delta.h deltacmd.h descendants.h diff.h diffcmd.h doc.h encode.h event.h file.h finfo.h graph.h http.h http_socket.h http_ssl.h http_transport.h info.h login.h main.h manifest.h md5.h merge.h merge3.h name.h pivot.h popen.h pqueue.h printf.h rebuild.h report.h rss.h schema.h search.h setup.h sha1.h shun.h skins.h stat.h style.h sync.h tag.h th_main.h timeline.h tkt.h tktsetup.h undo.h update.h url.h user.h verify.h vfile.h wiki.h wikiformat.h winhttp.h xfer.h zip.h
 
 page_index.h: $(TRANS_SRC) mkindex
 	./mkindex $(TRANS_SRC) >$@
 headers:	page_index.h makeheaders VERSION.h
-	./makeheaders  add_.c:add.h allrepo_.c:allrepo.h attach_.c:attach.h bag_.c:bag.h blob_.c:blob.h branch_.c:branch.h browse_.c:browse.h captcha_.c:captcha.h cgi_.c:cgi.h checkin_.c:checkin.h checkout_.c:checkout.h clearsign_.c:clearsign.h clone_.c:clone.h comformat_.c:comformat.h configure_.c:configure.h content_.c:content.h db_.c:db.h delta_.c:delta.h deltacmd_.c:deltacmd.h descendants_.c:descendants.h diff_.c:diff.h diffcmd_.c:diffcmd.h doc_.c:doc.h encode_.c:encode.h file_.c:file.h finfo_.c:finfo.h graph_.c:graph.h http_.c:http.h http_socket_.c:http_socket.h http_ssl_.c:http_ssl.h http_transport_.c:http_transport.h info_.c:info.h login_.c:login.h main_.c:main.h manifest_.c:manifest.h md5_.c:md5.h merge_.c:merge.h merge3_.c:merge3.h name_.c:name.h pivot_.c:pivot.h pqueue_.c:pqueue.h printf_.c:printf.h rebuild_.c:rebuild.h report_.c:report.h rss_.c:rss.h schema_.c:schema.h search_.c:search.h setup_.c:setup.h sha1_.c:sha1.h shun_.c:shun.h skins_.c:skins.h stat_.c:stat.h style_.c:style.h sync_.c:sync.h tag_.c:tag.h th_main_.c:th_main.h timeline_.c:timeline.h tkt_.c:tkt.h tktsetup_.c:tktsetup.h undo_.c:undo.h update_.c:update.h url_.c:url.h user_.c:user.h verify_.c:verify.h vfile_.c:vfile.h wiki_.c:wiki.h wikiformat_.c:wikiformat.h winhttp_.c:winhttp.h xfer_.c:xfer.h zip_.c:zip.h $(SRCDIR)/sqlite3.h $(SRCDIR)/th.h VERSION.h
+	./makeheaders  add_.c:add.h allrepo_.c:allrepo.h attach_.c:attach.h bag_.c:bag.h blob_.c:blob.h branch_.c:branch.h browse_.c:browse.h captcha_.c:captcha.h cgi_.c:cgi.h checkin_.c:checkin.h checkout_.c:checkout.h clearsign_.c:clearsign.h clone_.c:clone.h comformat_.c:comformat.h configure_.c:configure.h content_.c:content.h db_.c:db.h delta_.c:delta.h deltacmd_.c:deltacmd.h descendants_.c:descendants.h diff_.c:diff.h diffcmd_.c:diffcmd.h doc_.c:doc.h encode_.c:encode.h event_.c:event.h file_.c:file.h finfo_.c:finfo.h graph_.c:graph.h http_.c:http.h http_socket_.c:http_socket.h http_ssl_.c:http_ssl.h http_transport_.c:http_transport.h info_.c:info.h login_.c:login.h main_.c:main.h manifest_.c:manifest.h md5_.c:md5.h merge_.c:merge.h merge3_.c:merge3.h name_.c:name.h pivot_.c:pivot.h popen_.c:popen.h pqueue_.c:pqueue.h printf_.c:printf.h rebuild_.c:rebuild.h report_.c:report.h rss_.c:rss.h schema_.c:schema.h search_.c:search.h setup_.c:setup.h sha1_.c:sha1.h shun_.c:shun.h skins_.c:skins.h stat_.c:stat.h style_.c:style.h sync_.c:sync.h tag_.c:tag.h th_main_.c:th_main.h timeline_.c:timeline.h tkt_.c:tkt.h tktsetup_.c:tktsetup.h undo_.c:undo.h update_.c:update.h url_.c:url.h user_.c:user.h verify_.c:verify.h vfile_.c:vfile.h wiki_.c:wiki.h wikiformat_.c:wikiformat.h winhttp_.c:winhttp.h xfer_.c:xfer.h zip_.c:zip.h $(SRCDIR)/sqlite3.h $(SRCDIR)/th.h VERSION.h
 	touch headers
 headers: Makefile
 Makefile:
 add_.c:	$(SRCDIR)/add.c translate
 	./translate $(SRCDIR)/add.c >add_.c
@@ -447,10 +455,17 @@
 
 $(OBJDIR)/encode.o:	encode_.c encode.h  $(SRCDIR)/config.h
 	$(XTCC) -o $(OBJDIR)/encode.o -c encode_.c
 
 encode.h:	headers
+event_.c:	$(SRCDIR)/event.c translate
+	./translate $(SRCDIR)/event.c >event_.c
+
+$(OBJDIR)/event.o:	event_.c event.h  $(SRCDIR)/config.h
+	$(XTCC) -o $(OBJDIR)/event.o -c event_.c
+
+event.h:	headers
 file_.c:	$(SRCDIR)/file.c translate
 	./translate $(SRCDIR)/file.c >file_.c
 
 $(OBJDIR)/file.o:	file_.c file.h  $(SRCDIR)/config.h
 	$(XTCC) -o $(OBJDIR)/file.o -c file_.c
@@ -559,10 +574,17 @@
 
 $(OBJDIR)/pivot.o:	pivot_.c pivot.h  $(SRCDIR)/config.h
 	$(XTCC) -o $(OBJDIR)/pivot.o -c pivot_.c
 
 pivot.h:	headers
+popen_.c:	$(SRCDIR)/popen.c translate
+	./translate $(SRCDIR)/popen.c >popen_.c
+
+$(OBJDIR)/popen.o:	popen_.c popen.h  $(SRCDIR)/config.h
+	$(XTCC) -o $(OBJDIR)/popen.o -c popen_.c
+
+popen.h:	headers
 pqueue_.c:	$(SRCDIR)/pqueue.c translate
 	./translate $(SRCDIR)/pqueue.c >pqueue_.c
 
 $(OBJDIR)/pqueue.o:	pqueue_.c pqueue.h  $(SRCDIR)/config.h
 	$(XTCC) -o $(OBJDIR)/pqueue.o -c pqueue_.c
@@ -772,11 +794,14 @@
 
 zip.h:	headers
 $(OBJDIR)/sqlite3.o:	$(SRCDIR)/sqlite3.c
 	$(XTCC) -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 -c $(SRCDIR)/sqlite3.c -o $(OBJDIR)/sqlite3.o
 
+$(OBJDIR)/shell.o:	$(SRCDIR)/shell.c
+	$(XTCC) -Dmain=sqlite3_shell -DSQLITE_OMIT_LOAD_EXTENSION=1 -c $(SRCDIR)/shell.c -o $(OBJDIR)/shell.o
+
 $(OBJDIR)/th.o:	$(SRCDIR)/th.c
 	$(XTCC) -I$(SRCDIR) -c $(SRCDIR)/th.c -o $(OBJDIR)/th.o
 
 $(OBJDIR)/th_lang.o:	$(SRCDIR)/th_lang.c
 	$(XTCC) -I$(SRCDIR) -c $(SRCDIR)/th_lang.c -o $(OBJDIR)/th_lang.o
 

Index: src/makeheaders.c
==================================================================
--- src/makeheaders.c
+++ src/makeheaders.c
@@ -13,14 +13,17 @@
 #include 
 #include 
 #include 
 #include 
 #include 
-#ifndef WIN32
-# include 
-#else
+#if defined( __MINGW32__) ||  defined(__DMC__) || defined(_MSC_VER) || defined(__POCC__)
+#  ifndef WIN32
+#    define WIN32
+#  endif
 # include 
+#else
+# include 
 #endif
 
 /*
 ** Macros for debugging.
 */
@@ -2141,11 +2144,11 @@
     zArg = &zCmd[2];
     while( *zArg && isspace(*zArg) && *zArg!='\n' ){
       zArg++;
     }
     if( *zArg==0 || *zArg=='\n' ){ return 0; }
-    nArg = pToken->nText + (int)pToken->zText - (int)zArg;
+    nArg = pToken->nText + (int)(pToken->zText - zArg);
     if( nArg==9 && strncmp(zArg,"INTERFACE",9)==0 ){
       PushIfMacro(0,0,0,pToken->nLine,PS_Interface);
     }else if( nArg==16 && strncmp(zArg,"EXPORT_INTERFACE",16)==0 ){
       PushIfMacro(0,0,0,pToken->nLine,PS_Export);
     }else if( nArg==15 && strncmp(zArg,"LOCAL_INTERFACE",15)==0 ){
@@ -2160,11 +2163,11 @@
     zArg = &zCmd[5];
     while( *zArg && isspace(*zArg) && *zArg!='\n' ){
       zArg++;
     }
     if( *zArg==0 || *zArg=='\n' ){ return 0; }
-    nArg = pToken->nText + (int)pToken->zText - (int)zArg;
+    nArg = pToken->nText + (int)(pToken->zText - zArg);
     PushIfMacro("defined",zArg,nArg,pToken->nLine,0);
   }else if( nCmd==6 && strncmp(zCmd,"ifndef",6)==0 ){
     /*
     ** Push an #ifndef.
     */
@@ -2171,11 +2174,11 @@
     zArg = &zCmd[6];
     while( *zArg && isspace(*zArg) && *zArg!='\n' ){
       zArg++;
     }
     if( *zArg==0 || *zArg=='\n' ){ return 0; }
-    nArg = pToken->nText + (int)pToken->zText - (int)zArg;
+    nArg = pToken->nText + (int)(pToken->zText - zArg);
     PushIfMacro("!defined",zArg,nArg,pToken->nLine,0);
   }else if( nCmd==4 && strncmp(zCmd,"else",4)==0 ){
     /*
     ** Invert the #if on the top of the stack 
     */
@@ -2797,11 +2800,11 @@
   }else if( strncmp(zOldVersion,zTopLine,nTopLine)!=0 ){
     if( report ) fprintf(report,"error!\n");
     fprintf(stderr,
        "%s: Can't overwrite this file because it wasn't previously\n"
        "%*s  generated by 'makeheaders'.\n",
-       pFile->zHdr, strlen(pFile->zHdr), "");
+       pFile->zHdr, (int)strlen(pFile->zHdr), "");
     nErr++;
   }else if( strcmp(zOldVersion,zNewVersion)!=0 ){
     if( report ) fprintf(report,"updated\n");
     if( WriteFile(pFile->zHdr,zNewVersion) ){
       fprintf(stderr,"%s: Can't write to file\n",pFile->zHdr);
@@ -2952,18 +2955,18 @@
     if( nLabel==0 ) continue;
     zLabel[nLabel] = 0;
     InsertExtraDecl(pDecl);
     zDecl = pDecl->zDecl;
     if( zDecl==0 ) zDecl = pDecl->zFwd;
-    printf("%s %s %s %d %d %d %d %d %d\n",
+    printf("%s %s %s %p %d %d %d %d %d\n",
        pDecl->zName,
        zLabel,
        pDecl->zFile,
-       pDecl->pComment ? (int)pDecl->pComment/sizeof(Token) : 0,
+       pDecl->pComment,
        pDecl->pComment ? pDecl->pComment->nText+1 : 0,
-       pDecl->zIf ? strlen(pDecl->zIf)+1 : 0,
-       zDecl ? strlen(zDecl) : 0,
+       pDecl->zIf ? (int)strlen(pDecl->zIf)+1 : 0,
+       zDecl ? (int)strlen(zDecl) : 0,
        pDecl->pComment ? pDecl->pComment->nLine : 0,
        pDecl->tokenCode.nText ? pDecl->tokenCode.nText+1 : 0
     );
     if( pDecl->pComment ){
       printf("%.*s\n",pDecl->pComment->nText, pDecl->pComment->zText);

Index: src/makemake.tcl
==================================================================
--- src/makemake.tcl
+++ src/makemake.tcl
@@ -29,10 +29,11 @@
   descendants
   diff
   diffcmd
   doc
   encode
+  event
   file
   finfo
   graph
   http
   http_socket
@@ -44,10 +45,11 @@
   md5
   merge
   merge3
   name
   pivot
+  popen
   pqueue
   printf
   rebuild
   report
   rss
@@ -80,11 +82,11 @@
 }
 
 # Name of the final application
 #
 set name fossil
-
+if { 0 == $argc } {
 puts {# DO NOT EDIT
 #
 # This file is automatically generated.  Instead of editing this
 # file, edit "makemake.tcl" then run "tclsh makemake.tcl >main.mk"
 # to regenerate this file.
@@ -146,12 +148,18 @@
 		$(SRCDIR)/../manifest.uuid >>VERSION.h
 	awk '$$1=="D"{printf "#define MANIFEST_DATE \"%s %s\"\n",\
 		substr($$2,1,10),substr($$2,12)}' \
 		$(SRCDIR)/../manifest >>VERSION.h
 
-$(APPNAME):	headers $(OBJ) $(OBJDIR)/sqlite3.o $(OBJDIR)/th.o $(OBJDIR)/th_lang.o
-	$(TCC) -o $(APPNAME) $(OBJ) $(OBJDIR)/sqlite3.o $(OBJDIR)/th.o $(OBJDIR)/th_lang.o $(LIB)
+EXTRAOBJ = \
+  $(OBJDIR)/sqlite3.o \
+  $(OBJDIR)/shell.o \
+  $(OBJDIR)/th.o \
+  $(OBJDIR)/th_lang.o
+
+$(APPNAME):	headers $(OBJ) $(EXTRAOBJ)
+	$(TCC) -o $(APPNAME) $(OBJ) $(EXTRAOBJ) $(LIB)
 
 # This rule prevents make from using its default rules to try build
 # an executable named "manifest" out of the file named "manifest.c"
 #
 $(SRCDIR)/../manifest:	
@@ -199,10 +207,256 @@
 #append opt " -DSQLITE_ENABLE_FTS3=1"
 append opt " -Dlocaltime=fossil_localtime"
 append opt " -DSQLITE_ENABLE_LOCKING_STYLE=0"
 puts "\t\$(XTCC) $opt -c \$(SRCDIR)/sqlite3.c -o \$(OBJDIR)/sqlite3.o\n"
 
+puts "\$(OBJDIR)/shell.o:\t\$(SRCDIR)/shell.c"
+set opt {-Dmain=sqlite3_shell}
+append opt " -DSQLITE_OMIT_LOAD_EXTENSION=1"
+puts "\t\$(XTCC) $opt -c \$(SRCDIR)/shell.c -o \$(OBJDIR)/shell.o\n"
+
 puts "\$(OBJDIR)/th.o:\t\$(SRCDIR)/th.c"
 puts "\t\$(XTCC) -I\$(SRCDIR) -c \$(SRCDIR)/th.c -o \$(OBJDIR)/th.o\n"
 
 puts "\$(OBJDIR)/th_lang.o:\t\$(SRCDIR)/th_lang.c"
 puts "\t\$(XTCC) -I\$(SRCDIR) -c \$(SRCDIR)/th_lang.c -o \$(OBJDIR)/th_lang.o\n"
+exit
+}
+if { "dmc" == [lindex $argv 0] } {
+
+puts {# DO NOT EDIT
+#
+# This file is automatically generated.  Instead of editing this
+# file, edit "makemake.tcl" then run
+# "tclsh src/makemake.tcl dmc > win/Makefile.dmc"
+# to regenerate this file.
+B      = ..
+SRCDIR = $B\src
+OBJDIR = .
+O      = .obj
+E      = .exe
+
+
+# Maybe DMDIR, SSL or INCL needs adjustment
+DMDIR  = c:\DM
+INCL   = -I. -I$(SRCDIR) -I$B\win\include -I$(DMDIR)\extra\include
+
+#SSL   =  -DFOSSIL_ENABLE_SSL=1
+SSL    =
+
+DMCDEF =  -Dstrncasecmp=memicmp -Dstrcasecmp=stricmp
+I18N   =  -DFOSSIL_I18N=0
+
+CFLAGS = -o 
+BCC    = $(DMDIR)\bin\dmc $(CFLAGS)
+TCC    = $(DMDIR)\bin\dmc $(CFLAGS) $(DMCDEF) $(I18N) $(SSL) $(INCL)
+LIBS   = $(DMDIR)\extra\lib\ zlib wsock32
+}
+puts -nonewline "SRC   = "
+foreach s [lsort $src] {
+  puts -nonewline "${s}_.c "
+}
+puts "\n"
+puts -nonewline "OBJ   = "
+foreach s [lsort $src] {
+  puts -nonewline "\$(OBJDIR)\\$s\$O "
+}
+puts "\$(OBJDIR)\\sqlite3\$O \$(OBJDIR)\\th\$O \$(OBJDIR)\\th_lang\$O "
+puts {
+
+APPNAME = $(OBJDIR)\fossil$(E)
+
+all: $(APPNAME)
+
+$(APPNAME) : translate$E mkindex$E headers  $(OBJ) $(OBJDIR)\link
+	cd $(OBJDIR) 
+	$(DMDIR)\bin\link @link
+
+$(OBJDIR)\link: $B\win\Makefile.dmc}
+puts -nonewline "\t+echo "
+foreach s [lsort $src] {
+  puts -nonewline "$s "
+}
+puts "sqlite3 th th_lang > \$@"
+puts "\t+echo fossil >> \$@"
+puts "\t+echo fossil >> \$@"
+puts "\t+echo \$(LIBS) >> \$@\n\n"
+
+puts {
+translate$E: $(SRCDIR)\translate.c
+	$(BCC) -o$@ $**
+
+makeheaders$E: $(SRCDIR)\makeheaders.c
+	$(BCC) -o$@ $**
+
+mkindex$E: $(SRCDIR)\mkindex.c
+	$(BCC) -o$@ $**
+
+version$E: $B\win\version.c
+	$(BCC) -o$@ $**
+
+$(OBJDIR)\sqlite3$O : $(SRCDIR)\sqlite3.c
+	$(TCC) -o$@ -c -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 $**
+
+$(OBJDIR)\th$O : $(SRCDIR)\th.c
+	$(TCC) -o$@ -c $**
+
+$(OBJDIR)\th_lang$O : $(SRCDIR)\th_lang.c
+	$(TCC) -o$@ -c $**
+
+VERSION.h : version$E $B\manifest.uuid $B\manifest
+	+$** > $@
+
+page_index.h: mkindex$E $(SRC) 
+	+$** > $@
+
+clean:
+	-del $(OBJDIR)\*.obj
+	-del *.obj *_.c *.h *.map
+
+realclean:
+	-del $(APPNAME) translate$E mkindex$E makeheaders$E version$E
+
+}
+foreach s [lsort $src] {
+  puts "\$(OBJDIR)\\$s\$O : ${s}_.c ${s}.h"
+  puts "\t\$(TCC) -o\$@ -c ${s}_.c\n"
+  puts "${s}_.c : \$(SRCDIR)\\$s.c"
+  puts "\t+translate\$E \$** > \$@\n"
+}
+
+puts -nonewline "headers: makeheaders\$E page_index.h VERSION.h\n\t +makeheaders\$E "
+foreach s [lsort $src] {
+  puts -nonewline "${s}_.c:$s.h "
+}
+puts "\$(SRCDIR)\\sqlite3.h \$(SRCDIR)\\th.h VERSION.h"
+puts "\t@copy /Y nul: headers"
+exit
+}
+
+if { "msc" == [lindex $argv 0] } {
+
+puts {# DO NOT EDIT
+#
+# This file is automatically generated.  Instead of editing this
+# file, edit "makemake.tcl" then run
+# "tclsh src/makemake.tcl msc > win/Makefile.msc"
+# to regenerate this file.
+B      = ..
+SRCDIR = $B\src
+OBJDIR = .
+O      = .obj
+E      = .exe
+
+# Maybe MSCDIR, SSL, ZLIB, or INCL needs adjustment
+MSCDIR = c:\msc
+
+# Uncomment below for SSL support
+SSL =
+SSLLIB =
+#SSL = -DFOSSIL_ENABLE_SSL=1
+#SSLLIB  = ssleay32.lib libeay32.lib user32.lib gdi32.lib advapi32.lib
+
+# zlib options
+# When using precompiled from http://zlib.net/zlib125-dll.zip
+#ZINCDIR = C:\zlib125-dll\include
+#ZLIBDIR = C:\zlib125-dll\lib
+#ZLIB    = zdll.lib
+ZINCDIR = $(MSCDIR)\extra\include
+ZLIBDIR = $(MSCDIR)\extra\lib
+ZLIB    = zlib.lib
+
+INCL   = -I. -I$(SRCDIR) -I$B\win\include -I$(MSCDIR)\extra\include -I$(ZINCDIR)
+
+MSCDEF =  -Dstrncasecmp=memicmp -Dstrcasecmp=stricmp
+I18N   =  -DFOSSIL_I18N=0
+
+CFLAGS = -nologo -MT -O2
+BCC    = $(CC) $(CFLAGS)
+TCC    = $(CC) -c $(CFLAGS) $(MSCDEF) $(I18N) $(SSL) $(INCL)
+LIBS   = $(ZLIB) ws2_32.lib $(SSLLIB)
+LIBDIR = -LIBPATH:$(MSCDIR)\extra\lib -LIBPATH:$(ZLIBDIR)
+}
+puts -nonewline "SRC   = "
+foreach s [lsort $src] {
+  puts -nonewline "${s}_.c "
+}
+puts "\n"
+puts -nonewline "OBJ   = "
+foreach s [lsort $src] {
+  puts -nonewline "\$(OBJDIR)\\$s\$O "
+}
+puts "\$(OBJDIR)\\sqlite3\$O \$(OBJDIR)\\th\$O \$(OBJDIR)\\th_lang\$O "
+puts {
+
+APPNAME = $(OBJDIR)\fossil$(E)
+
+all: $(OBJDIR) $(APPNAME)
+
+$(APPNAME) : translate$E mkindex$E headers $(OBJ) $(OBJDIR)\linkopts
+	cd $(OBJDIR) 
+	link -LINK -OUT:$@ $(LIBDIR) @linkopts
+
+$(OBJDIR)\linkopts: $B\win\Makefile.msc}
+puts -nonewline "\techo "
+foreach s [lsort $src] {
+  puts -nonewline "$s "
+}
+puts "sqlite3 th th_lang > \$@"
+puts "\techo \$(LIBS) >> \$@\n\n"
+
+puts {
+
+$(OBJDIR):
+	@-mkdir $@
+
+translate$E: $(SRCDIR)\translate.c
+	$(BCC) $**
+
+makeheaders$E: $(SRCDIR)\makeheaders.c
+	$(BCC) $**
+
+mkindex$E: $(SRCDIR)\mkindex.c
+	$(BCC) $**
+
+version$E: $B\win\version.c
+	$(BCC) $**
+
+$(OBJDIR)\sqlite3$O : $(SRCDIR)\sqlite3.c
+	$(TCC) /Fo$@ -c -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 $**
+
+$(OBJDIR)\th$O : $(SRCDIR)\th.c
+	$(TCC) /Fo$@ -c $**
+
+$(OBJDIR)\th_lang$O : $(SRCDIR)\th_lang.c
+	$(TCC) /Fo$@ -c $**
+
+VERSION.h : version$E $B\manifest.uuid $B\manifest
+	$** > $@
+
+page_index.h: mkindex$E $(SRC) 
+	$** > $@
+
+clean:
+	-del $(OBJDIR)\*.obj
+	-del *.obj *_.c *.h *.map
+	-del headers linkopts
+
+realclean:
+	-del $(APPNAME) translate$E mkindex$E makeheaders$E version$E
+
+}
+foreach s [lsort $src] {
+  puts "\$(OBJDIR)\\$s\$O : ${s}_.c ${s}.h"
+  puts "\t\$(TCC) /Fo\$@ -c ${s}_.c\n"
+  puts "${s}_.c : \$(SRCDIR)\\$s.c"
+  puts "\ttranslate\$E \$** > \$@\n"
+}
+
+puts -nonewline "headers: makeheaders\$E page_index.h VERSION.h\n\tmakeheaders\$E "
+foreach s [lsort $src] {
+  puts -nonewline "${s}_.c:$s.h "
+}
+puts "\$(SRCDIR)\\sqlite3.h \$(SRCDIR)\\th.h VERSION.h"
+puts "\t@copy /Y nul: headers"
+	
+}

ADDED   src/makeskins.c
Index: src/makeskins.c
==================================================================
--- src/makeskins.c
+++ src/makeskins.c
@@ -0,0 +1,53 @@
+/*
+** Copyright (c) 2010 Michael T. Richter, assigned to the Fossil SCM project.
+**
+** This program is free software; you can redistribute it and/or
+** modify it under the terms of the Simplified BSD License (also
+** known as the "2-Clause License" or "FreeBSD License".)
+**
+** This program is distributed in the hope that it will be useful,
+** but without any warranty; without even the implied warranty of
+** merchantability or fitness for a particular purpose.
+**
+** Author contact information:
+**   ttmrichter@gmail.com
+**
+*******************************************************************************
+**
+** This utility is a preprocessor that takes raw CSS and HTML files, along with
+** plain text descriptions, and generates the skins.c file that is used to embed
+** the distributed default skins into Fossil.
+**
+** The intent of the utility is to make adding new skins to a Fossil build
+** easier without involving error-prone C programming.
+**
+** This utility must be run BEFORE the translate program is executed on the C
+** source files.  (This is in retrospect obvious since it generates one of the
+** said C files.)
+*/
+#include 
+#include 
+#include 
+#include 
+
+int main(int argc, char **argv){
+  if( argc==3 ){
+    /*
+    ** 1. Validate the arguments:  and  directory.
+    ** 2. Open the /skins.c file.
+    ** 3. Write the initial boilerplate.
+    ** 4. Walk the directory structure of .
+    ** 5. For each one:
+    **    a) Store the title and author information. (info.txt)
+    **    b) Write out the description as a comment. (info.txt)
+    **    c) Write out the CSS information.          (style.css)
+    **    d) Write out the header information.       (header.html)
+    **    e) Write out the footer information.       (footer.html)
+    ** 6. Write out the built-in skins table.
+    ** 7. Write the trailing boilerplate.
+    */
+  }else{
+    /* error -- need a pair of directories */
+  }
+  return 0;
+}

Index: src/manifest.c
==================================================================
--- src/manifest.c
+++ src/manifest.c
@@ -26,42 +26,55 @@
 
 #if INTERFACE
 /*
 ** Types of control files
 */
+#define CFTYPE_ANY        0
 #define CFTYPE_MANIFEST   1
 #define CFTYPE_CLUSTER    2
 #define CFTYPE_CONTROL    3
 #define CFTYPE_WIKI       4
 #define CFTYPE_TICKET     5
 #define CFTYPE_ATTACHMENT 6
+#define CFTYPE_EVENT      7
+
+/*
+** A single F-card within a manifest
+*/
+struct ManifestFile { 
+  char *zName;           /* Name of a file */
+  char *zUuid;           /* UUID of the file */
+  char *zPerm;           /* File permissions */
+  char *zPrior;          /* Prior name if the name was changed */
+};
+
 
 /*
 ** A parsed manifest or cluster.
 */
 struct Manifest {
   Blob content;         /* The original content blob */
   int type;             /* Type of artifact.  One of CFTYPE_xxxxx */
+  int rid;              /* The blob-id for this manifest */
+  char *zBaseline;      /* Baseline manifest.  The B card. */
+  Manifest *pBaseline;  /* The actual baseline manifest */
   char *zComment;       /* Decoded comment.  The C card. */
   double rDate;         /* Date and time from D card.  0.0 if no D card. */
   char *zUser;          /* Name of the user from the U card. */
   char *zRepoCksum;     /* MD5 checksum of the baseline content.  R card. */
   char *zWiki;          /* Text of the wiki page.  W card. */
   char *zWikiTitle;     /* Name of the wiki page. L card. */
+  double rEventDate;    /* Date of an event.  E card. */
+  char *zEventId;       /* UUID for an event.  E card. */
   char *zTicketUuid;    /* UUID for a ticket. K card. */
   char *zAttachName;    /* Filename of an attachment. A card. */
   char *zAttachSrc;     /* UUID of document being attached. A card. */
   char *zAttachTarget;  /* Ticket or wiki that attachment applies to.  A card */
   int nFile;            /* Number of F cards */
   int nFileAlloc;       /* Slots allocated in aFile[] */
-  struct { 
-    char *zName;           /* Name of a file */
-    char *zUuid;           /* UUID of the file */
-    char *zPerm;           /* File permissions */
-    char *zPrior;          /* Prior name if the name was changed */
-    int iRename;           /* index of renamed name in prior/next manifest */
-  } *aFile;             /* One entry for each F card */
+  int iFile;            /* Index of current file in iterator */
+  ManifestFile *aFile;  /* One entry for each F-card */
   int nParent;          /* Number of parents. */
   int nParentAlloc;     /* Slots allocated in azParent[] */
   char **azParent;      /* UUIDs of parents.  One for each P card argument */
   int nCChild;          /* Number of cluster children */
   int nCChildAlloc;     /* Number of closts allocated in azCChild[] */
@@ -80,22 +93,192 @@
     char *zValue;          /* Value of the field */
   } *aField;            /* One for each J card */
 };
 #endif
 
+/*
+** A cache of parsed manifests.  This reduces the number of
+** calls to manifest_parse() when doing a rebuild.
+*/
+#define MX_MANIFEST_CACHE 6
+static struct {
+  int nxAge;
+  int aAge[MX_MANIFEST_CACHE];
+  Manifest *apManifest[MX_MANIFEST_CACHE];
+} manifestCache;
+
 
 /*
 ** Clear the memory allocated in a manifest object
 */
-void manifest_clear(Manifest *p){
-  blob_reset(&p->content);
-  free(p->aFile);
-  free(p->azParent);
-  free(p->azCChild);
-  free(p->aTag);
-  free(p->aField);
-  memset(p, 0, sizeof(*p));
+void manifest_destroy(Manifest *p){
+  if( p ){
+    blob_reset(&p->content);
+    free(p->aFile);
+    free(p->azParent);
+    free(p->azCChild);
+    free(p->aTag);
+    free(p->aField);
+    if( p->pBaseline ) manifest_destroy(p->pBaseline);
+    fossil_free(p);
+  }
+}
+
+/*
+** Add an element to the manifest cache using LRU replacement.
+*/
+void manifest_cache_insert(Manifest *p){
+  while( p ){
+    int i;
+    Manifest *pBaseline = p->pBaseline;
+    p->pBaseline = 0;
+    for(i=0; i=MX_MANIFEST_CACHE ){
+      int oldest = 0;
+      int oldestAge = manifestCache.aAge[0];
+      for(i=1; irid==rid ){
+      p = manifestCache.apManifest[i];
+      manifestCache.apManifest[i] = 0;
+      return p;
+    }
+  }
+  return 0;
+}
+
+/*
+** Clear the manifest cache.
+*/
+void manifest_cache_clear(void){
+  int i;
+  for(i=0; i=n ) return;
+  z += i;
+  n -= i;
+  *pz = z;
+  for(i=n-1; i>=0; i--){
+    if( z[i]=='\n' && memcmp(&z[i],"\n-----BEGIN PGP SIGNATURE-", 25)==0 ){
+      n = i+1;
+      break;
+    }
+  }
+  *pn = n;
+  return;
+}
+
+/*
+** Verify the Z-card checksum on the artifact, if there is such a
+** checksum.  Return 0 if there is no Z-card.  Return 1 if the Z-card
+** exists and is correct.  Return 2 if the Z-card exists and has the wrong
+** value.
+**
+**   0123456789 123456789 123456789 123456789 
+**   Z aea84f4f863865a8d59d0384e4d2a41c
+*/
+static int verify_z_card(const char *z, int n){
+  if( n<35 ) return 0;
+  if( z[n-35]!='Z' || z[n-34]!=' ' ) return 0;
+  md5sum_init();
+  md5sum_step_text(z, n-35);
+  if( memcmp(&z[n-33], md5sum_finish(0), 32)==0 ){
+    return 1;
+  }else{
+    return 2;
+  }
+}
+
+/*
+** A structure used for rapid parsing of the Manifest file
+*/
+typedef struct ManifestText ManifestText;
+struct ManifestText {
+  char *z;           /* The first character of the next token */
+  char *zEnd;        /* One character beyond the end of the manifest */
+  int atEol;         /* True if z points to the start of a new line */
+};
+
+/*
+** Return a pointer to the next token.  The token is zero-terminated.
+** Return NULL if there are no more tokens on the current line.
+*/
+static char *next_token(ManifestText *p, int *pLen){
+  char *z;
+  char *zStart;
+  int c;
+  if( p->atEol ) return 0;
+  zStart = z = p->z;
+  while( (c=(*z))!=' ' && c!='\n' ){ z++; }
+  *z = 0;
+  p->z = &z[1];
+  p->atEol = c=='\n';
+  if( pLen ) *pLen = z - zStart;
+  return zStart;
+}
+
+/*
+** Return the card-type for the next card.  Or, return 0 if there are no
+** more cards or if we are not at the end of the current card.
+*/
+static char next_card(ManifestText *p){
+  char c;
+  if( !p->atEol || p->z>=p->zEnd ) return 0;
+  c = p->z[0];
+  if( p->z[1]==' ' ){
+    p->z += 2;
+    p->atEol = 0;
+  }else if( p->z[1]=='\n' ){
+    p->z += 2;
+    p->atEol = 1;
+  }else{
+    c = 0;
+  }
+  return c;
 }
 
 /*
 ** Parse a blob into a Manifest object.  The Manifest object
 ** takes over the input blob and will free it when the
@@ -121,48 +304,66 @@
 ** Each card is divided into tokens by a single space character.
 ** The first token is a single upper-case letter which is the card type.
 ** The card type determines the other parameters to the card.
 ** Cards must occur in lexicographical order.
 */
-int manifest_parse(Manifest *p, Blob *pContent){
-  int seenHeader = 0;
+static Manifest *manifest_parse(Blob *pContent, int rid){
+  Manifest *p;
   int seenZ = 0;
   int i, lineNo=0;
-  Blob line, token, a1, a2, a3, a4;
+  ManifestText x;
   char cPrevType = 0;
+  char cType;
+  char *z;
+  int n;
+  char *zUuid;
+  int sz = 0;
+
+  /* Every control artifact ends with a '\n' character.  Exit early
+  ** if that is not the case for this artifact.
+  */
+  z = blob_buffer(pContent);
+  n = blob_size(pContent);
+  if( n<=0 || z[n-1]!='\n' ){
+    blob_reset(pContent);
+    return 0;
+  }
+
+  /* Strip off the PGP signature if there is one.  Then verify the
+  ** Z-card.
+  */
+  remove_pgp_signature(&z, &n);
+  if( verify_z_card(z, n)==0 ){
+    blob_reset(pContent);
+    return 0;
+  }
+
+  /* Verify that the first few characters of the artifact look like
+  ** a control artifact.
+  */
+  if( n<10 || z[0]<'A' || z[0]>'Z' || z[1]!=' ' ){
+    blob_reset(pContent);
+    return 0;
+  }
 
+  /* Allocate a Manifest object to hold the parsed control artifact.
+  */
+  p = fossil_malloc( sizeof(*p) );
   memset(p, 0, sizeof(*p));
   memcpy(&p->content, pContent, sizeof(p->content));
+  p->rid = rid;
   blob_zero(pContent);
   pContent = &p->content;
 
-  blob_zero(&a1);
-  blob_zero(&a2);
-  blob_zero(&a3);
-  md5sum_init();
-  while( blob_line(pContent, &line) ){
-    char *z = blob_buffer(&line);
+  /* Begin parsing, card by card.
+  */
+  x.z = z;
+  x.zEnd = &z[n];
+  x.atEol = 1;
+  while( (cType = next_card(&x))!=0 && cType>=cPrevType ){
     lineNo++;
-    if( z[0]=='-' ){
-      if( strncmp(z, "-----BEGIN PGP ", 15)!=0 ){
-        goto manifest_syntax_error;
-      }
-      if( seenHeader ){
-        break;
-      }
-      while( blob_line(pContent, &line)>2 ){}
-      if( blob_line(pContent, &line)==0 ) break;
-      z = blob_buffer(&line);
-    }
-    if( z[0]  ??
       **
       ** Identifies an attachment to either a wiki page or a ticket.
       **  is the artifact that is the attachment.  
@@ -169,50 +370,60 @@
       ** is omitted to delete an attachment.   is the name of
       ** a wiki page or ticket to which that attachment is connected.
       */
       case 'A': {
         char *zName, *zTarget, *zSrc;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)==0 ) goto manifest_syntax_error;
+        int nTarget = 0, nSrc = 0;
+        zName = next_token(&x, 0);
+        zTarget = next_token(&x, &nTarget);
+        zSrc = next_token(&x, &nSrc);
+        if( zName==0 || zTarget==0 ) goto manifest_syntax_error;      
         if( p->zAttachName!=0 ) goto manifest_syntax_error;
-        zName = blob_terminate(&a1);
-        zTarget = blob_terminate(&a2);
-        blob_token(&line, &a3);
-        zSrc = blob_terminate(&a3);
         defossilize(zName);
         if( !file_is_simple_pathname(zName) ){
           goto manifest_syntax_error;
         }
         defossilize(zTarget);
-        if( (blob_size(&a2)!=UUID_SIZE || !validate16(zTarget, UUID_SIZE))
+        if( (nTarget!=UUID_SIZE || !validate16(zTarget, UUID_SIZE))
            && !wiki_name_is_wellformed((const unsigned char *)zTarget) ){
           goto manifest_syntax_error;
         }
-        if( blob_size(&a3)>0
-         && (blob_size(&a3)!=UUID_SIZE || !validate16(zSrc, UUID_SIZE)) ){
+        if( zSrc && (nSrc!=UUID_SIZE || !validate16(zSrc, UUID_SIZE)) ){
           goto manifest_syntax_error;
         }
         p->zAttachName = (char*)file_tail(zName);
         p->zAttachSrc = zSrc;
         p->zAttachTarget = zTarget;
         break;
       }
+
+      /*
+      **    B 
+      **
+      ** A B-line gives the UUID for the baselinen of a delta-manifest.
+      */
+      case 'B': {
+        if( p->zBaseline ) goto manifest_syntax_error;
+        p->zBaseline = next_token(&x, &sz);
+        if( p->zBaseline==0 ) goto manifest_syntax_error;
+        if( sz!=UUID_SIZE ) goto manifest_syntax_error;
+        if( !validate16(p->zBaseline, UUID_SIZE) ) goto manifest_syntax_error;
+        break;
+      }
+
 
       /*
       **     C 
       **
       ** Comment text is fossil-encoded.  There may be no more than
       ** one C line.  C lines are required for manifests and are
       ** disallowed on all other control files.
       */
       case 'C': {
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
         if( p->zComment!=0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        p->zComment = blob_terminate(&a1);
+        p->zComment = next_token(&x, 0);
+        if( p->zComment==0 ) goto manifest_syntax_error;
         defossilize(p->zComment);
         break;
       }
 
       /*
@@ -221,63 +432,73 @@
       ** The timestamp should be ISO 8601.   YYYY-MM-DDtHH:MM:SS
       ** There can be no more than 1 D line.  D lines are required
       ** for all control files except for clusters.
       */
       case 'D': {
-        char *zDate;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( p->rDate!=0.0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        zDate = blob_terminate(&a1);
-        p->rDate = db_double(0.0, "SELECT julianday(%Q)", zDate);
+        if( p->rDate>0.0 ) goto manifest_syntax_error;
+        p->rDate = db_double(0.0, "SELECT julianday(%Q)", next_token(&x,0));
+        if( p->rDate<=0.0 ) goto manifest_syntax_error;
+        break;
+      }
+
+      /*
+      **     E  
+      **
+      ** An "event" card that contains the timestamp of the event in the 
+      ** format YYYY-MM-DDtHH:MM:SS and a unique identifier for the event.
+      ** The event timestamp is distinct from the D timestamp.  The D
+      ** timestamp is when the artifact was created whereas the E timestamp
+      ** is when the specific event is said to occur.
+      */
+      case 'E': {
+        if( p->rEventDate>0.0 ) goto manifest_syntax_error;
+        p->rEventDate = db_double(0.0,"SELECT julianday(%Q)", next_token(&x,0));
+        if( p->rEventDate<=0.0 ) goto manifest_syntax_error;
+        p->zEventId = next_token(&x, &sz);
+        if( sz!=UUID_SIZE ) goto manifest_syntax_error;
+        if( !validate16(p->zEventId, UUID_SIZE) ) goto manifest_syntax_error;
         break;
       }
 
       /*
-      **     F   ?? ??
+      **     F  ?? ?? ??
       **
       ** Identifies a file in a manifest.  Multiple F lines are
       ** allowed in a manifest.  F lines are not allowed in any
       ** other control file.  The filename and old-name are fossil-encoded.
       */
       case 'F': {
-        char *zName, *zUuid, *zPerm, *zPriorName;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)==0 ) goto manifest_syntax_error;
-        zName = blob_terminate(&a1);
-        zUuid = blob_terminate(&a2);
-        blob_token(&line, &a3);
-        zPerm = blob_terminate(&a3);
-        if( blob_size(&a2)!=UUID_SIZE ) goto manifest_syntax_error;
-        if( !validate16(zUuid, UUID_SIZE) ) goto manifest_syntax_error;
+        char *zName, *zPerm, *zPriorName;
+        zName = next_token(&x,0);
+        if( zName==0 ) goto manifest_syntax_error;
         defossilize(zName);
         if( !file_is_simple_pathname(zName) ){
           goto manifest_syntax_error;
         }
-        blob_token(&line, &a4);
-        zPriorName = blob_terminate(&a4);
-        if( zPriorName[0] ){
+        zUuid = next_token(&x, &sz);
+        if( p->zBaseline==0 || zUuid!=0 ){
+          if( sz!=UUID_SIZE ) goto manifest_syntax_error;
+          if( !validate16(zUuid, UUID_SIZE) ) goto manifest_syntax_error;
+        }
+        zPerm = next_token(&x,0);
+        zPriorName = next_token(&x,0);
+        if( zPriorName ){
           defossilize(zPriorName);
           if( !file_is_simple_pathname(zPriorName) ){
             goto manifest_syntax_error;
           }
-        }else{
-          zPriorName = 0;
         }
         if( p->nFile>=p->nFileAlloc ){
           p->nFileAlloc = p->nFileAlloc*2 + 10;
-          p->aFile = realloc(p->aFile, p->nFileAlloc*sizeof(p->aFile[0]) );
-          if( p->aFile==0 ) fossil_panic("out of memory");
+          p->aFile = fossil_realloc(p->aFile, 
+                                    p->nFileAlloc*sizeof(p->aFile[0]) );
         }
         i = p->nFile++;
         p->aFile[i].zName = zName;
         p->aFile[i].zUuid = zUuid;
         p->aFile[i].zPerm = zPerm;
         p->aFile[i].zPrior = zPriorName;
-        p->aFile[i].iRename = -1;
         if( i>0 && strcmp(p->aFile[i-1].zName, zName)>=0 ){
           goto manifest_syntax_error;
         }
         break;
       }
@@ -290,22 +511,19 @@
       ** value.  If  is omitted then it is understood to be an
       ** empty string.
       */
       case 'J': {
         char *zName, *zValue;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        blob_token(&line, &a2);
-        if( blob_token(&line, &a3)!=0 ) goto manifest_syntax_error;
-        zName = blob_terminate(&a1);
-        zValue = blob_terminate(&a2);
+        zName = next_token(&x,0);
+        zValue = next_token(&x,0);
+        if( zName==0 ) goto manifest_syntax_error;
+        if( zValue==0 ) zValue = "";
         defossilize(zValue);
         if( p->nField>=p->nFieldAlloc ){
           p->nFieldAlloc = p->nFieldAlloc*2 + 10;
-          p->aField = realloc(p->aField,
+          p->aField = fossil_realloc(p->aField,
                                p->nFieldAlloc*sizeof(p->aField[0]) );
-          if( p->aField==0 ) fossil_panic("out of memory");
         }
         i = p->nField++;
         p->aField[i].zName = zName;
         p->aField[i].zValue = zValue;
         if( i>0 && strcmp(p->aField[i-1].zName, zName)>=0 ){
@@ -320,18 +538,14 @@
       **
       ** A K-line gives the UUID for the ticket which this control file
       ** is amending.
       */
       case 'K': {
-        char *zUuid;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        zUuid = blob_terminate(&a1);
-        if( blob_size(&a1)!=UUID_SIZE ) goto manifest_syntax_error;
-        if( !validate16(zUuid, UUID_SIZE) ) goto manifest_syntax_error;
         if( p->zTicketUuid!=0 ) goto manifest_syntax_error;
-        p->zTicketUuid = zUuid;
+        p->zTicketUuid = next_token(&x, &sz);
+        if( sz!=UUID_SIZE ) goto manifest_syntax_error;
+        if( !validate16(p->zTicketUuid, UUID_SIZE) ) goto manifest_syntax_error;
         break;
       }
 
       /*
       **     L 
@@ -338,15 +552,13 @@
       **
       ** The wiki page title is fossil-encoded.  There may be no more than
       ** one L line.
       */
       case 'L': {
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
         if( p->zWikiTitle!=0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        p->zWikiTitle = blob_terminate(&a1);
+        p->zWikiTitle = next_token(&x,0);
+        if( p->zWikiTitle==0 ) goto manifest_syntax_error;
         defossilize(p->zWikiTitle);
         if( !wiki_name_is_wellformed((const unsigned char *)p->zWikiTitle) ){
           goto manifest_syntax_error;
         }
         break;
@@ -357,21 +569,18 @@
       **
       ** An M-line identifies another artifact by its UUID.  M-lines
       ** occur in clusters only.
       */
       case 'M': {
-        char *zUuid;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        zUuid = blob_terminate(&a1);
-        if( blob_size(&a1)!=UUID_SIZE ) goto manifest_syntax_error;
+        zUuid = next_token(&x, &sz);
+        if( zUuid==0 ) goto manifest_syntax_error;
+        if( sz!=UUID_SIZE ) goto manifest_syntax_error;
         if( !validate16(zUuid, UUID_SIZE) ) goto manifest_syntax_error;
         if( p->nCChild>=p->nCChildAlloc ){
           p->nCChildAlloc = p->nCChildAlloc*2 + 10;
-          p->azCChild = 
-             realloc(p->azCChild, p->nCChildAlloc*sizeof(p->azCChild[0]) );
-          if( p->azCChild==0 ) fossil_panic("out of memory");
+          p->azCChild = fossil_realloc(p->azCChild
+                                 , p->nCChildAlloc*sizeof(p->azCChild[0]) );
         }
         i = p->nCChild++;
         p->azCChild[i] = zUuid;
         if( i>0 && strcmp(p->azCChild[i-1], zUuid)>=0 ){
           goto manifest_syntax_error;
@@ -385,20 +594,17 @@
       ** Specify one or more other artifacts where are the parents of
       ** this artifact.  The first parent is the primary parent.  All
       ** others are parents by merge.
       */
       case 'P': {
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        while( blob_token(&line, &a1) ){
-          char *zUuid;
-          if( blob_size(&a1)!=UUID_SIZE ) goto manifest_syntax_error;
-          zUuid = blob_terminate(&a1);
+        while( (zUuid = next_token(&x, &sz))!=0 ){
+          if( sz!=UUID_SIZE ) goto manifest_syntax_error;
           if( !validate16(zUuid, UUID_SIZE) ) goto manifest_syntax_error;
           if( p->nParent>=p->nParentAlloc ){
             p->nParentAlloc = p->nParentAlloc*2 + 5;
-            p->azParent = realloc(p->azParent, p->nParentAlloc*sizeof(char*));
-            if( p->azParent==0 ) fossil_panic("out of memory");
+            p->azParent = fossil_realloc(p->azParent,
+                               p->nParentAlloc*sizeof(char*));
           }
           i = p->nParent++;
           p->azParent[i] = zUuid;
         }
         break;
@@ -405,20 +611,17 @@
       }
 
       /*
       **     R 
       **
-      ** Specify the MD5 checksum of the entire baseline in a
-      ** manifest.
+      ** Specify the MD5 checksum over the name and content of all files
+      ** in the manifest.
       */
       case 'R': {
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
         if( p->zRepoCksum!=0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        if( blob_size(&a1)!=32 ) goto manifest_syntax_error;
-        p->zRepoCksum = blob_terminate(&a1);
+        p->zRepoCksum = next_token(&x, &sz);
+        if( sz!=32 ) goto manifest_syntax_error;
         if( !validate16(p->zRepoCksum, 32) ) goto manifest_syntax_error;
         break;
       }
 
       /*
@@ -435,29 +638,20 @@
       ** the tag is really a property with the given value.
       **
       ** Tags are not allowed in clusters.  Multiple T lines are allowed.
       */
       case 'T': {
-        char *zName, *zUuid, *zValue;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ){
-          goto manifest_syntax_error;
-        }
-        if( blob_token(&line, &a2)==0 ){
-          goto manifest_syntax_error;
-        }
-        zName = blob_terminate(&a1);
-        zUuid = blob_terminate(&a2);
-        if( blob_token(&line, &a3)==0 ){
-          zValue = 0;
-        }else{
-          zValue = blob_terminate(&a3);
-          defossilize(zValue);
-        }
-        if( blob_size(&a2)==UUID_SIZE && validate16(zUuid, UUID_SIZE) ){
+        char *zName, *zValue;
+        zName = next_token(&x, 0);
+        if( zName==0 ) goto manifest_syntax_error;
+        zUuid = next_token(&x, &sz);
+        if( zUuid==0 ) goto manifest_syntax_error;
+        zValue = next_token(&x, 0);
+        if( zValue ) defossilize(zValue);
+        if( sz==UUID_SIZE && validate16(zUuid, UUID_SIZE) ){
           /* A valid uuid */
-        }else if( blob_size(&a2)==1 && zUuid[0]=='*' ){
+        }else if( sz==1 && zUuid[0]=='*' ){
           zUuid = 0;
         }else{
           goto manifest_syntax_error;
         }
         defossilize(zName);
@@ -468,12 +662,11 @@
           /* Do not allow tags whose names look like UUIDs */
           goto manifest_syntax_error;
         }
         if( p->nTag>=p->nTagAlloc ){
           p->nTagAlloc = p->nTagAlloc*2 + 10;
-          p->aTag = realloc(p->aTag, p->nTagAlloc*sizeof(p->aTag[0]) );
-          if( p->aTag==0 ) fossil_panic("out of memory");
+          p->aTag = fossil_realloc(p->aTag, p->nTagAlloc*sizeof(p->aTag[0]) );
         }
         i = p->nTag++;
         p->aTag[i].zName = zName;
         p->aTag[i].zUuid = zUuid;
         p->aTag[i].zValue = zValue;
@@ -489,19 +682,17 @@
       ** Identify the user who created this control file by their
       ** login.  Only one U line is allowed.  Prohibited in clusters.
       ** If the user name is omitted, take that to be "anonymous".
       */
       case 'U': {
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
         if( p->zUser!=0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a1)==0 ){
+        p->zUser = next_token(&x, 0);
+        if( p->zUser==0 ){
           p->zUser = "anonymous";
         }else{
-          p->zUser = blob_terminate(&a1);
           defossilize(p->zUser);
         }
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
         break;
       }
 
       /*
       **     W 
@@ -509,26 +700,28 @@
       ** The next  bytes of the file contain the text of the wiki
       ** page.  There is always an extra \n before the start of the next
       ** record.
       */
       case 'W': {
-        int size;
+        char *zSize;
+        int size, c;
         Blob wiki;
-        md5sum_step_text(blob_buffer(&line), blob_size(&line));
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        if( !blob_is_int(&a1, &size) ) goto manifest_syntax_error;
+        zSize = next_token(&x, 0);
+        if( zSize==0 ) goto manifest_syntax_error;
+        if( x.atEol==0 ) goto manifest_syntax_error;
+        for(size=0; (c = zSize[0])>='0' && c<='9'; zSize++){
+           size = size*10 + c - '0';
+        }
         if( size<0 ) goto manifest_syntax_error;
         if( p->zWiki!=0 ) goto manifest_syntax_error;
         blob_zero(&wiki);
-        if( blob_extract(pContent, size+1, &wiki)!=size+1 ){
-          goto manifest_syntax_error;
-        }
-        p->zWiki = blob_buffer(&wiki);
-        md5sum_step_text(p->zWiki, size+1);
-        if( p->zWiki[size]!='\n' ) goto manifest_syntax_error;
-        p->zWiki[size] = 0;
+        if( (&x.z[size+1])>=x.zEnd ) goto manifest_syntax_error;
+        p->zWiki = x.z;
+        x.z += size;
+        if( x.z[0]!='\n' ) goto manifest_syntax_error;
+        x.z[0] = 0;
+        x.z++;
         break;
       }
 
 
       /*
@@ -541,37 +734,31 @@
       ** This card is required for all control file types except for
       ** Manifest.  It is not required for manifest only for historical
       ** compatibility reasons.
       */
       case 'Z': {
-        int rc;
-        Blob hash;
-        if( blob_token(&line, &a1)==0 ) goto manifest_syntax_error;
-        if( blob_token(&line, &a2)!=0 ) goto manifest_syntax_error;
-        if( blob_size(&a1)!=32 ) goto manifest_syntax_error;
-        if( !validate16(blob_buffer(&a1), 32) ) goto manifest_syntax_error;
-        md5sum_finish(&hash);
-        rc = blob_compare(&hash, &a1);
-        blob_reset(&hash);
-        if( rc!=0 ) goto manifest_syntax_error;
+        zUuid = next_token(&x, &sz);
+        if( sz!=32 ) goto manifest_syntax_error;
+        if( !validate16(zUuid, 32) ) goto manifest_syntax_error;
         seenZ = 1;
         break;
       }
       default: {
         goto manifest_syntax_error;
       }
     }
   }
-  if( !seenHeader ) goto manifest_syntax_error;
+  if( x.znFile>0 || p->zRepoCksum!=0 ){
+  if( p->nFile>0 || p->zRepoCksum!=0 || p->zBaseline ){
     if( p->nCChild>0 ) goto manifest_syntax_error;
-    if( p->rDate==0.0 ) goto manifest_syntax_error;
+    if( p->rDate<=0.0 ) goto manifest_syntax_error;
     if( p->nField>0 ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
     if( p->zWiki ) goto manifest_syntax_error;
     if( p->zWikiTitle ) goto manifest_syntax_error;
+    if( p->zEventId ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
     if( p->zAttachName ) goto manifest_syntax_error;
     p->type = CFTYPE_MANIFEST;
   }else if( p->nCChild>0 ){
     if( p->rDate>0.0 ) goto manifest_syntax_error;
@@ -581,26 +768,41 @@
     if( p->nParent>0 ) goto manifest_syntax_error;
     if( p->nField>0 ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
     if( p->zWiki ) goto manifest_syntax_error;
     if( p->zWikiTitle ) goto manifest_syntax_error;
+    if( p->zEventId ) goto manifest_syntax_error;
     if( p->zAttachName ) goto manifest_syntax_error;
     if( !seenZ ) goto manifest_syntax_error;
     p->type = CFTYPE_CLUSTER;
   }else if( p->nField>0 ){
-    if( p->rDate==0.0 ) goto manifest_syntax_error;
+    if( p->rDate<=0.0 ) goto manifest_syntax_error;
     if( p->zWiki ) goto manifest_syntax_error;
     if( p->zWikiTitle ) goto manifest_syntax_error;
+    if( p->zEventId ) goto manifest_syntax_error;
     if( p->nCChild>0 ) goto manifest_syntax_error;
     if( p->nTag>0 ) goto manifest_syntax_error;
     if( p->zTicketUuid==0 ) goto manifest_syntax_error;
     if( p->zUser==0 ) goto manifest_syntax_error;
     if( p->zAttachName ) goto manifest_syntax_error;
     if( !seenZ ) goto manifest_syntax_error;
     p->type = CFTYPE_TICKET;
+  }else if( p->zEventId ){
+    if( p->rDate<=0.0 ) goto manifest_syntax_error;
+    if( p->nCChild>0 ) goto manifest_syntax_error;
+    if( p->zTicketUuid!=0 ) goto manifest_syntax_error;
+    if( p->zWikiTitle!=0 ) goto manifest_syntax_error;
+    if( p->zWiki==0 ) goto manifest_syntax_error;
+    if( p->zAttachName ) goto manifest_syntax_error;
+    for(i=0; inTag; i++){
+      if( p->aTag[i].zName[0]!='+' ) goto manifest_syntax_error;
+      if( p->aTag[i].zUuid!=0 ) goto manifest_syntax_error;
+    }
+    if( !seenZ ) goto manifest_syntax_error;
+    p->type = CFTYPE_EVENT;
   }else if( p->zWiki!=0 ){
-    if( p->rDate==0.0 ) goto manifest_syntax_error;
+    if( p->rDate<=0.0 ) goto manifest_syntax_error;
     if( p->nCChild>0 ) goto manifest_syntax_error;
     if( p->nTag>0 ) goto manifest_syntax_error;
     if( p->zTicketUuid!=0 ) goto manifest_syntax_error;
     if( p->zWikiTitle==0 ) goto manifest_syntax_error;
     if( p->zAttachName ) goto manifest_syntax_error;
@@ -614,11 +816,11 @@
     if( p->zAttachName ) goto manifest_syntax_error;
     if( !seenZ ) goto manifest_syntax_error;
     p->type = CFTYPE_CONTROL;
   }else if( p->zAttachName ){
     if( p->nCChild>0 ) goto manifest_syntax_error;
-    if( p->rDate==0.0 ) goto manifest_syntax_error;
+    if( p->rDate<=0.0 ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
     if( p->zWikiTitle ) goto manifest_syntax_error;
     if( !seenZ ) goto manifest_syntax_error;
     p->type = CFTYPE_ATTACHMENT;
   }else{
@@ -625,43 +827,199 @@
     if( p->nCChild>0 ) goto manifest_syntax_error;
     if( p->rDate<=0.0 ) goto manifest_syntax_error;
     if( p->nParent>0 ) goto manifest_syntax_error;
     if( p->nField>0 ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
-    if( p->zWiki ) goto manifest_syntax_error;
     if( p->zWikiTitle ) goto manifest_syntax_error;
     if( p->zTicketUuid ) goto manifest_syntax_error;
-    if( p->zAttachName ) goto manifest_syntax_error;
     p->type = CFTYPE_MANIFEST;
   }
   md5sum_init();
-  return 1;
+  return p;
 
 manifest_syntax_error:
   /*fprintf(stderr, "Manifest error on line %i\n", lineNo);fflush(stderr);*/
   md5sum_init();
-  manifest_clear(p);
+  manifest_destroy(p);
   return 0;
 }
+
+/*
+** Get a manifest given the rid for the control artifact.  Return
+** a pointer to the manifest on success or NULL if there is a failure.
+*/
+Manifest *manifest_get(int rid, int cfType){
+  Blob content;
+  Manifest *p;
+  p = manifest_cache_find(rid);
+  if( p ){
+    if( cfType!=CFTYPE_ANY && cfType!=p->type ){
+      manifest_cache_insert(p);
+      p = 0;
+    }
+    return p;
+  }
+  content_get(rid, &content);
+  p = manifest_parse(&content, rid);
+  if( p && cfType!=CFTYPE_ANY && cfType!=p->type ){
+    manifest_destroy(p);
+    p = 0;
+  }
+  return p;
+}
+
+/*
+** Given a checkin name, load and parse the manifest for that checkin.
+** Throw a fatal error if anything goes wrong.
+*/
+Manifest *manifest_get_by_name(const char *zName, int *pRid){
+  int rid;
+  Manifest *p;
+
+  rid = name_to_rid(zName);
+  if( !is_a_version(rid) ){
+    fossil_fatal("no such checkin: %s", zName);
+  }
+  if( pRid ) *pRid = rid;
+  p = manifest_get(rid, CFTYPE_MANIFEST);
+  if( p==0 ){
+    fossil_fatal("cannot parse manifest for checkin: %s", zName);
+  }
+  return p;
+}
 
 /*
 ** COMMAND: test-parse-manifest
 **
-** Usage: %fossil test-parse-manifest FILENAME
+** Usage: %fossil test-parse-manifest FILENAME ?N?
 **
 ** Parse the manifest and discarded.  Use for testing only.
 */
 void manifest_test_parse_cmd(void){
-  Manifest m;
+  Manifest *p;
   Blob b;
-  if( g.argc!=3 ){
+  int i;
+  int n = 1;
+  sqlite3_open(":memory:", &g.db);
+  if( g.argc!=3 && g.argc!=4 ){
     usage("FILENAME");
   }
-  db_must_be_within_tree();
   blob_read_from_file(&b, g.argv[2]);
-  manifest_parse(&m, &b);
-  manifest_clear(&m);
+  if( g.argc>3 ) n = atoi(g.argv[3]);
+  for(i=0; izBaseline!=0 && p->pBaseline==0 ){
+    int rid = uuid_to_rid(p->zBaseline, 0);
+    if( rid==0 && !throwError ){
+      rid = content_new(p->zBaseline);
+      db_multi_exec(
+         "INSERT OR IGNORE INTO orphan(rid, baseline) VALUES(%d,%d)",
+         rid, p->rid
+      );
+      return 1;
+    }
+    p->pBaseline = manifest_get(rid, CFTYPE_MANIFEST);
+    if( p->pBaseline==0 ){
+      if( !throwError && db_exists("SELECT 1 FROM phantom WHERE rid=%d",rid) ){
+        db_multi_exec(
+           "INSERT OR IGNORE INTO orphan(rid, baseline) VALUES(%d,%d)",
+           rid, p->rid
+        );
+        return 1;
+      }    
+      fossil_fatal("cannot access baseline manifest %S", p->zBaseline);
+    }
+  }
+  return 0;
+}
+
+/*
+** Rewind a manifest-file iterator back to the beginning of the manifest.
+*/
+void manifest_file_rewind(Manifest *p){
+  p->iFile = 0;
+  fetch_baseline(p, 1);
+  if( p->pBaseline ){
+    p->pBaseline->iFile = 0;
+  }
+}
+
+/*
+** Advance to the next manifest-file.
+**
+** Return NULL for end-of-records or if there is an error.  If an error
+** occurs and pErr!=0 then store 1 in *pErr.
+*/
+ManifestFile *manifest_file_next(
+  Manifest *p,   
+  int *pErr
+){
+  ManifestFile *pOut = 0;
+  if( pErr ) *pErr = 0;
+  if( p->pBaseline==0 ){
+    /* Manifest p is a baseline-manifest.  Just scan down the list
+    ** of files. */
+    if( p->iFilenFile ) pOut = &p->aFile[p->iFile++];
+  }else{
+    /* Manifest p is a delta-manifest.  Scan the baseline but amend the
+    ** file list in the baseline with changes described by p.
+    */
+    Manifest *pB = p->pBaseline;
+    int cmp;
+    while(1){
+      if( pB->iFile>=pB->nFile ){
+        /* We have used all entries out of the baseline.  Return the next
+        ** entry from the delta. */
+        if( p->iFilenFile ) pOut = &p->aFile[p->iFile++];
+        break;
+      }else if( p->iFile>=p->nFile ){
+        /* We have used all entries from the delta.  Return the next
+        ** entry from the baseline. */
+        if( pB->iFilenFile ) pOut = &pB->aFile[pB->iFile++];
+        break;
+      }else if( (cmp = strcmp(pB->aFile[pB->iFile].zName,
+                              p->aFile[p->iFile].zName)) < 0 ){
+        /* The next baseline entry comes before the next delta entry.
+        ** So return the baseline entry. */
+        pOut = &pB->aFile[pB->iFile++];
+        break;
+      }else if( cmp>0 ){
+        /* The next delta entry comes before the next baseline
+        ** entry so return the delta entry */
+        pOut = &p->aFile[p->iFile++];
+        break;
+      }else if( p->aFile[p->iFile].zUuid ){
+        /* The next delta entry is a replacement for the next baseline
+        ** entry.  Skip the baseline entry and return the delta entry */
+        pB->iFile++;
+        pOut = &p->aFile[p->iFile++];
+        break;
+      }else{
+        /* The next delta entry is a delete of the next baseline
+        ** entry.  Skip them both.  Repeat the loop to find the next
+        ** non-delete entry. */
+        pB->iFile++;
+        p->iFile++;
+        continue;
+      }
+    }
+  }
+  return pOut;
 }
 
 /*
 ** Translate a filename into a filename-id (fnid).  Create a new fnid
 ** if no previously exists.
@@ -689,14 +1047,14 @@
 ** Add a single entry to the mlink table.  Also add the filename to
 ** the filename table if it is not there already.
 */
 static void add_one_mlink(
   int mid,                  /* The record ID of the manifest */
-  const char *zFromUuid,    /* UUID for the mlink.pid field */
-  const char *zToUuid,      /* UUID for the mlink.fid field */
+  const char *zFromUuid,    /* UUID for the mlink.pid. "" to add file */
+  const char *zToUuid,      /* UUID for the mlink.fid. "" to delele */
   const char *zFilename,    /* Filename */
-  const char *zPrior        /* Previous filename.  NULL if unchanged */
+  const char *zPrior        /* Previous filename. NULL if unchanged */
 ){
   int fnid, pfnid, pid, fid;
   static Stmt s1;
 
   fnid = filename_to_fnid(zFilename);
@@ -703,16 +1061,16 @@
   if( zPrior==0 ){
     pfnid = 0;
   }else{
     pfnid = filename_to_fnid(zPrior);
   }
-  if( zFromUuid==0 ){
+  if( zFromUuid==0 || zFromUuid[0]==0 ){
     pid = 0;
   }else{
     pid = uuid_to_rid(zFromUuid, 1);
   }
-  if( zToUuid==0 ){
+  if( zToUuid==0 || zToUuid[0]==0 ){
     fid = 0;
   }else{
     fid = uuid_to_rid(zToUuid, 1);
   }
   db_static_prepare(&s1,
@@ -729,33 +1087,83 @@
     content_deltify(pid, fid, 0);
   }
 }
 
 /*
-** Locate a file named zName in the aFile[] array of the given
-** manifest.  We assume that filenames are in sorted order.
-** Use a binary search.  Return turn the index of the matching
-** entry.  Or return -1 if not found.
+** Do a binary search to find a file in the p->aFile[] array.  
+**
+** As an optimization, guess that the file we seek is at index p->iFile.
+** That will usually be the case.  If it is not found there, then do the
+** actual binary search.
+**
+** Update p->iFile to be the index of the file that is found.
 */
-static int find_file_in_manifest(Manifest *p, const char *zName){
+static ManifestFile *manifest_file_seek_base(Manifest *p, const char *zName){
   int lwr, upr;
   int c;
   int i;
   lwr = 0;
   upr = p->nFile - 1;
+  if( p->iFile>=lwr && p->iFileaFile[p->iFile+1].zName, zName);
+    if( c==0 ){
+      return &p->aFile[++p->iFile];
+    }else if( c>0 ){
+      upr = p->iFile;
+    }else{
+      lwr = p->iFile+1;
+    }
+  }
   while( lwr<=upr ){
     i = (lwr+upr)/2;
     c = strcmp(p->aFile[i].zName, zName);
     if( c<0 ){
       lwr = i+1;
     }else if( c>0 ){
       upr = i-1;
     }else{
-      return i;
+      p->iFile = i;
+      return &p->aFile[i];
     }
   }
-  return -1;
+  return 0;
+}
+
+/*
+** Locate a file named zName in the aFile[] array of the given manifest.
+** Return a pointer to the appropriate ManifestFile object.  Return NULL
+** if not found.
+**
+** This routine works even if p is a delta-manifest.  The pointer 
+** returned might be to the baseline.
+**
+** We assume that filenames are in sorted order and use a binary search.
+*/
+ManifestFile *manifest_file_seek(Manifest *p, const char *zName){
+  ManifestFile *pFile;
+  
+  pFile = manifest_file_seek_base(p, zName);
+  if( pFile && pFile->zUuid==0 ) return 0;
+  if( pFile==0 && p->zBaseline ){
+    fetch_baseline(p, 1);
+    pFile = manifest_file_seek_base(p->pBaseline, zName);
+  }
+  return pFile;
+}
+
+/*
+** This strcmp() function handles NULL arguments.  NULLs sort first.
+*/
+static int strcmp_null(const char *zOne, const char *zTwo){
+  if( zOne==0 ){
+    if( zTwo==0 ) return 0;
+    return -1;
+  }else if( zTwo==0 ){
+    return +1;
+  }else{
+    return strcmp(zOne, zTwo);
+  }
 }
 
 /*
 ** Add mlink table entries associated with manifest cid.  The
 ** parent manifest is pid.
@@ -766,85 +1174,76 @@
 ** Deleted files have mlink.fid=0.
 ** Added files have mlink.pid=0.
 ** Edited files have both mlink.pid!=0 and mlink.fid!=0
 */
 static void add_mlink(int pid, Manifest *pParent, int cid, Manifest *pChild){
-  Manifest other;
   Blob otherContent;
-  int i, j;
+  int otherRid;
+  int i, rc;
+  ManifestFile *pChildFile, *pParentFile;
+  Manifest **ppOther;
+  static Stmt eq;
 
-  if( db_exists("SELECT 1 FROM mlink WHERE mid=%d", cid) ){
-    return;
-  }
+  db_static_prepare(&eq, "SELECT 1 FROM mlink WHERE mid=:mid");
+  db_bind_int(&eq, ":mid", cid);
+  rc = db_step(&eq);
+  db_reset(&eq);
+  if( rc==SQLITE_ROW ) return;
+
   assert( pParent==0 || pChild==0 );
   if( pParent==0 ){
-    pParent = &other;
-    content_get(pid, &otherContent);
-  }else{
-    pChild = &other;
-    content_get(cid, &otherContent);
-  }
-  if( blob_size(&otherContent)==0 ) return;
-  if( manifest_parse(&other, &otherContent)==0 ) return;
-  content_deltify(pid, cid, 0);
-
-  /* Use the iRename fields to find the cross-linkage between
-  ** renamed files.  */
-  for(j=0; jnFile; j++){
-    const char *zPrior = pChild->aFile[j].zPrior;
-    if( zPrior && zPrior[0] ){
-      i = find_file_in_manifest(pParent, zPrior);
-      if( i>=0 ){
-        pChild->aFile[j].iRename = i;
-        pParent->aFile[i].iRename = j;
-      }
-    }
-  }
-
-  /* Construct the mlink entries */
-  for(i=j=0; inFile && jnFile; ){
-    int c;
-    if( pParent->aFile[i].iRename>=0 ){
-      i++;
-    }else if( (c = strcmp(pParent->aFile[i].zName, pChild->aFile[j].zName))<0 ){
-      add_one_mlink(cid, pParent->aFile[i].zUuid,0,pParent->aFile[i].zName,0);
-      i++;
-    }else if( c>0 ){
-      int rn = pChild->aFile[j].iRename;
-      if( rn>=0 ){
-        add_one_mlink(cid, pParent->aFile[rn].zUuid, pChild->aFile[j].zUuid, 
-                      pChild->aFile[j].zName, pParent->aFile[rn].zName);
-      }else{
-        add_one_mlink(cid, 0, pChild->aFile[j].zUuid, pChild->aFile[j].zName,0);
-      }
-      j++;
-    }else{
-      if( strcmp(pParent->aFile[i].zUuid, pChild->aFile[j].zUuid)!=0 ){
-        add_one_mlink(cid, pParent->aFile[i].zUuid, pChild->aFile[j].zUuid, 
-                      pChild->aFile[j].zName, 0);
-      }
-      i++;
-      j++;
-    }
-  }
-  while( inFile ){
-    if( pParent->aFile[i].iRename<0 ){
-      add_one_mlink(cid, pParent->aFile[i].zUuid, 0, pParent->aFile[i].zName,0);
-    }
-    i++;
-  }
-  while( jnFile ){
-    int rn = pChild->aFile[j].iRename;
-    if( rn>=0 ){
-      add_one_mlink(cid, pParent->aFile[rn].zUuid, pChild->aFile[j].zUuid, 
-                    pChild->aFile[j].zName, pParent->aFile[rn].zName);
-    }else{
-      add_one_mlink(cid, 0, pChild->aFile[j].zUuid, pChild->aFile[j].zName,0);
-    }
-    j++;
-  }
-  manifest_clear(&other);
+    ppOther = &pParent;
+    otherRid = pid;
+  }else{
+    ppOther = &pChild;
+    otherRid = cid;
+  }
+  if( (*ppOther = manifest_cache_find(otherRid))==0 ){
+    content_get(otherRid, &otherContent);
+    if( blob_size(&otherContent)==0 ) return;
+    *ppOther = manifest_parse(&otherContent, otherRid);
+    if( *ppOther==0 ) return;
+  }
+  if( fetch_baseline(pParent, 0) || fetch_baseline(pChild, 0) ){
+    manifest_destroy(*ppOther);
+    return;
+  }
+  if( (pParent->zBaseline==0)==(pChild->zBaseline==0) ){
+    content_deltify(pid, cid, 0); 
+  }else if( pChild->zBaseline==0 && pParent->zBaseline!=0 ){
+    content_deltify(pParent->pBaseline->rid, cid, 0);
+  }
+  
+  for(i=0, pChildFile=pChild->aFile; inFile; i++, pChildFile++){
+    if( pChildFile->zPrior ){
+       pParentFile = manifest_file_seek(pParent, pChildFile->zPrior);
+       if( pParentFile ){
+         add_one_mlink(cid, pParentFile->zUuid, pChildFile->zUuid,
+                       pChildFile->zName, pChildFile->zPrior);
+       }
+    }else{
+       pParentFile = manifest_file_seek(pParent, pChildFile->zName);
+       if( pParentFile==0 ){
+         if( pChildFile->zUuid ){
+           add_one_mlink(cid, 0, pChildFile->zUuid, pChildFile->zName, 0);
+         }
+       }else if( strcmp_null(pChildFile->zUuid, pParentFile->zUuid)!=0 ){
+         add_one_mlink(cid, pParentFile->zUuid, pChildFile->zUuid,
+                       pChildFile->zName, 0);
+       }
+    }
+  }
+  if( pParent->zBaseline && pChild->zBaseline ){
+    for(i=0, pParentFile=pParent->aFile; inFile; i++, pParentFile++){
+      if( pParentFile->zUuid ) continue;
+      pChildFile = manifest_file_seek(pChild, pParentFile->zName);
+      if( pChildFile ){
+        add_one_mlink(cid, 0, pChildFile->zUuid, pChildFile->zName, 0);
+      }
+    }
+  }
+  manifest_cache_insert(*ppOther);
 }
 
 /*
 ** True if manifest_crosslink_begin() has been called but
 ** manifest_crosslink_end() is still pending.
@@ -964,10 +1363,12 @@
 **      *  Manifest
 **      *  Control
 **      *  Wiki Page
 **      *  Ticket Change
 **      *  Cluster
+**      *  Attachment
+**      *  Event
 **
 ** If the input is a control artifact, then make appropriate entries
 ** in the auxiliary tables of the database in order to crosslink the
 ** artifact.
 **
@@ -979,38 +1380,44 @@
 ** of the routine, "manifest_crosslink", and the name of this source
 ** file, is a legacy of its original use.
 */
 int manifest_crosslink(int rid, Blob *pContent){
   int i;
-  Manifest m;
+  Manifest *p;
   Stmt q;
   int parentid = 0;
 
-  if( manifest_parse(&m, pContent)==0 ){
+  if( (p = manifest_cache_find(rid))!=0 ){
+    blob_reset(pContent);
+  }else if( (p = manifest_parse(pContent, rid))==0 ){
+    return 0;
+  }
+  if( g.xlinkClusterOnly && p->type!=CFTYPE_CLUSTER ){
+    manifest_destroy(p);
     return 0;
   }
-  if( g.xlinkClusterOnly && m.type!=CFTYPE_CLUSTER ){
-    manifest_clear(&m);
+  if( p->type==CFTYPE_MANIFEST && fetch_baseline(p, 0) ){
+    manifest_destroy(p);
     return 0;
   }
   db_begin_transaction();
-  if( m.type==CFTYPE_MANIFEST ){
+  if( p->type==CFTYPE_MANIFEST ){
     if( !db_exists("SELECT 1 FROM mlink WHERE mid=%d", rid) ){
       char *zCom;
-      for(i=0; inParent; i++){
+        int pid = uuid_to_rid(p->azParent[i], 1);
         db_multi_exec("INSERT OR IGNORE INTO plink(pid, cid, isprim, mtime)"
-                      "VALUES(%d, %d, %d, %.17g)", pid, rid, i==0, m.rDate);
+                      "VALUES(%d, %d, %d, %.17g)", pid, rid, i==0, p->rDate);
         if( i==0 ){
-          add_mlink(pid, 0, rid, &m);
+          add_mlink(pid, 0, rid, p);
           parentid = pid;
         }
       }
       db_prepare(&q, "SELECT cid FROM plink WHERE pid=%d AND isprim", rid);
       while( db_step(&q)==SQLITE_ROW ){
         int cid = db_column_int(&q, 0);
-        add_mlink(rid, &m, cid, 0);
+        add_mlink(rid, p, cid, 0);
       }
       db_finalize(&q);
       db_multi_exec(
         "REPLACE INTO event(type,mtime,objid,user,comment,"
                            "bgcolor,euser,ecomment)"
@@ -1021,179 +1428,226 @@
         "  ),"
         "  %d,%Q,%Q,"
         "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d AND tagtype>0),"
         "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d),"
         "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d));",
-        TAG_DATE, rid, m.rDate,
-        rid, m.zUser, m.zComment, 
+        TAG_DATE, rid, p->rDate,
+        rid, p->zUser, p->zComment, 
         TAG_BGCOLOR, rid,
         TAG_USER, rid,
         TAG_COMMENT, rid
       );
       zCom = db_text(0, "SELECT coalesce(ecomment, comment) FROM event"
                         " WHERE rowid=last_insert_rowid()");
-      wiki_extract_links(zCom, rid, 0, m.rDate, 1, WIKI_INLINE);
+      wiki_extract_links(zCom, rid, 0, p->rDate, 1, WIKI_INLINE);
       free(zCom);
+
+      /* If this is a delta-manifest, record the fact that this repository
+      ** contains delta manifests, to free the "commit" logic to generate
+      ** new delta manifests.
+      */
+      if( p->zBaseline!=0 ){
+        static int once = 0;
+        if( !once ){
+          db_set_int("seen-delta-manifest", 1, 0);
+          once = 0;
+        }
+      }
     }
   }
-  if( m.type==CFTYPE_CLUSTER ){
-    tag_insert("cluster", 1, 0, rid, m.rDate, rid);
-    for(i=0; itype==CFTYPE_CLUSTER ){
+    static Stmt del1;
+    tag_insert("cluster", 1, 0, rid, p->rDate, rid);
+    db_static_prepare(&del1, "DELETE FROM unclustered WHERE rid=:rid");
+    for(i=0; inCChild; i++){
       int mid;
-      mid = uuid_to_rid(m.azCChild[i], 1);
+      mid = uuid_to_rid(p->azCChild[i], 1);
       if( mid>0 ){
-        db_multi_exec("DELETE FROM unclustered WHERE rid=%d", mid);
+        db_bind_int(&del1, ":rid", mid);
+        db_step(&del1);
+        db_reset(&del1);
       }
     }
   }
-  if( m.type==CFTYPE_CONTROL || m.type==CFTYPE_MANIFEST ){
-    for(i=0; itype==CFTYPE_CONTROL
+   || p->type==CFTYPE_MANIFEST
+   || p->type==CFTYPE_EVENT
+  ){
+    for(i=0; inTag; i++){
       int tid;
       int type;
-      if( m.aTag[i].zUuid ){
-        tid = uuid_to_rid(m.aTag[i].zUuid, 1);
+      if( p->aTag[i].zUuid ){
+        tid = uuid_to_rid(p->aTag[i].zUuid, 1);
       }else{
         tid = rid;
       }
       if( tid ){
-        switch( m.aTag[i].zName[0] ){
+        switch( p->aTag[i].zName[0] ){
           case '-':  type = 0;  break;  /* Cancel prior occurances */
           case '+':  type = 1;  break;  /* Apply to target only */
           case '*':  type = 2;  break;  /* Propagate to descendants */
           default:
-            fossil_fatal("unknown tag type in manifest: %s", m.aTag);
+            fossil_fatal("unknown tag type in manifest: %s", p->aTag);
             return 0;
         }
-        tag_insert(&m.aTag[i].zName[1], type, m.aTag[i].zValue, 
-                   rid, m.rDate, tid);
+        tag_insert(&p->aTag[i].zName[1], type, p->aTag[i].zValue, 
+                   rid, p->rDate, tid);
       }
     }
     if( parentid ){
       tag_propagate_all(parentid);
     }
   }
-  if( m.type==CFTYPE_WIKI ){
-    char *zTag = mprintf("wiki-%s", m.zWikiTitle);
+  if( p->type==CFTYPE_WIKI ){
+    char *zTag = mprintf("wiki-%s", p->zWikiTitle);
     int tagid = tag_findid(zTag, 1);
     int prior;
     char *zComment;
     int nWiki;
     char zLength[40];
-    while( isspace(m.zWiki[0]) ) m.zWiki++;
-    nWiki = strlen(m.zWiki);
+    while( fossil_isspace(p->zWiki[0]) ) p->zWiki++;
+    nWiki = strlen(p->zWiki);
     sqlite3_snprintf(sizeof(zLength), zLength, "%d", nWiki);
-    tag_insert(zTag, 1, zLength, rid, m.rDate, rid);
+    tag_insert(zTag, 1, zLength, rid, p->rDate, rid);
     free(zTag);
     prior = db_int(0,
       "SELECT rid FROM tagxref"
       " WHERE tagid=%d AND mtime<%.17g"
       " ORDER BY mtime DESC",
-      tagid, m.rDate
+      tagid, p->rDate
     );
     if( prior ){
       content_deltify(prior, rid, 0);
     }
     if( nWiki>0 ){
-      zComment = mprintf("Changes to wiki page [%h]", m.zWikiTitle);
+      zComment = mprintf("Changes to wiki page [%h]", p->zWikiTitle);
     }else{
-      zComment = mprintf("Deleted wiki page [%h]", m.zWikiTitle);
+      zComment = mprintf("Deleted wiki page [%h]", p->zWikiTitle);
     }
     db_multi_exec(
       "REPLACE INTO event(type,mtime,objid,user,comment,"
       "                  bgcolor,euser,ecomment)"
       "VALUES('w',%.17g,%d,%Q,%Q,"
       "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d AND tagtype>1),"
       "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d),"
       "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d));",
-      m.rDate, rid, m.zUser, zComment, 
+      p->rDate, rid, p->zUser, zComment, 
       TAG_BGCOLOR, rid,
       TAG_BGCOLOR, rid,
       TAG_USER, rid,
       TAG_COMMENT, rid
     );
     free(zComment);
   }
-  if( m.type==CFTYPE_TICKET ){
+  if( p->type==CFTYPE_EVENT ){
+    char *zTag = mprintf("event-%s", p->zEventId);
+    int tagid = tag_findid(zTag, 1);
+    int prior, subsequent;
+    int nWiki;
+    char zLength[40];
+    while( fossil_isspace(p->zWiki[0]) ) p->zWiki++;
+    nWiki = strlen(p->zWiki);
+    sqlite3_snprintf(sizeof(zLength), zLength, "%d", nWiki);
+    tag_insert(zTag, 1, zLength, rid, p->rDate, rid);
+    free(zTag);
+    prior = db_int(0,
+      "SELECT rid FROM tagxref"
+      " WHERE tagid=%d AND mtime<%.17g"
+      " ORDER BY mtime DESC",
+      tagid, p->rDate
+    );
+    if( prior ){
+      content_deltify(prior, rid, 0);
+      db_multi_exec(
+        "DELETE FROM event"
+        " WHERE type='e'"
+        "   AND tagid=%d"
+        "   AND objid IN (SELECT rid FROM tagxref WHERE tagid=%d)",
+        tagid, tagid
+      );
+    }
+    subsequent = db_int(0,
+      "SELECT rid FROM tagxref"
+      " WHERE tagid=%d AND mtime>%.17g"
+      " ORDER BY mtime",
+      tagid, p->rDate
+    );
+    if( subsequent ){
+      content_deltify(rid, subsequent, 0);
+    }else{
+      db_multi_exec(
+        "REPLACE INTO event(type,mtime,objid,tagid,user,comment,bgcolor)"
+        "VALUES('e',%.17g,%d,%d,%Q,%Q,"
+        "  (SELECT value FROM tagxref WHERE tagid=%d AND rid=%d));",
+        p->rEventDate, rid, tagid, p->zUser, p->zComment, 
+        TAG_BGCOLOR, rid
+      );
+    }
+  }
+  if( p->type==CFTYPE_TICKET ){
     char *zTag;
 
     assert( manifest_crosslink_busy==1 );
-    zTag = mprintf("tkt-%s", m.zTicketUuid);
-    tag_insert(zTag, 1, 0, rid, m.rDate, rid);
+    zTag = mprintf("tkt-%s", p->zTicketUuid);
+    tag_insert(zTag, 1, 0, rid, p->rDate, rid);
     free(zTag);
     db_multi_exec("INSERT OR IGNORE INTO pending_tkt VALUES(%Q)",
-                  m.zTicketUuid);
+                  p->zTicketUuid);
   }
-  if( m.type==CFTYPE_ATTACHMENT ){
+  if( p->type==CFTYPE_ATTACHMENT ){
     db_multi_exec(
        "INSERT INTO attachment(attachid, mtime, src, target,"
                                         "filename, comment, user)"
        "VALUES(%d,%.17g,%Q,%Q,%Q,%Q,%Q);",
-       rid, m.rDate, m.zAttachSrc, m.zAttachTarget, m.zAttachName,
-       (m.zComment ? m.zComment : ""), m.zUser
+       rid, p->rDate, p->zAttachSrc, p->zAttachTarget, p->zAttachName,
+       (p->zComment ? p->zComment : ""), p->zUser
     );
     db_multi_exec(
        "UPDATE attachment SET isLatest = (mtime=="
           "(SELECT max(mtime) FROM attachment"
           "  WHERE target=%Q AND filename=%Q))"
        " WHERE target=%Q AND filename=%Q",
-       m.zAttachTarget, m.zAttachName,
-       m.zAttachTarget, m.zAttachName
+       p->zAttachTarget, p->zAttachName,
+       p->zAttachTarget, p->zAttachName
     );
-    if( strlen(m.zAttachTarget)!=UUID_SIZE
-     || !validate16(m.zAttachTarget, UUID_SIZE) 
+    if( strlen(p->zAttachTarget)!=UUID_SIZE
+     || !validate16(p->zAttachTarget, UUID_SIZE) 
     ){
       char *zComment;
-      if( m.zAttachSrc && m.zAttachSrc[0] ){
+      if( p->zAttachSrc && p->zAttachSrc[0] ){
         zComment = mprintf("Add attachment \"%h\" to wiki page [%h]",
-             m.zAttachName, m.zAttachTarget);
+             p->zAttachName, p->zAttachTarget);
       }else{
         zComment = mprintf("Delete attachment \"%h\" from wiki page [%h]",
-             m.zAttachName, m.zAttachTarget);
+             p->zAttachName, p->zAttachTarget);
       }
       db_multi_exec(
         "REPLACE INTO event(type,mtime,objid,user,comment)"
         "VALUES('w',%.17g,%d,%Q,%Q)",
-        m.rDate, rid, m.zUser, zComment
+        p->rDate, rid, p->zUser, zComment
       );
       free(zComment);
     }else{
       char *zComment;
-      if( m.zAttachSrc && m.zAttachSrc[0] ){
+      if( p->zAttachSrc && p->zAttachSrc[0] ){
         zComment = mprintf("Add attachment \"%h\" to ticket [%.10s]",
-             m.zAttachName, m.zAttachTarget);
+             p->zAttachName, p->zAttachTarget);
       }else{
         zComment = mprintf("Delete attachment \"%h\" from ticket [%.10s]",
-             m.zAttachName, m.zAttachTarget);
+             p->zAttachName, p->zAttachTarget);
       }
       db_multi_exec(
         "REPLACE INTO event(type,mtime,objid,user,comment)"
         "VALUES('t',%.17g,%d,%Q,%Q)",
-        m.rDate, rid, m.zUser, zComment
+        p->rDate, rid, p->zUser, zComment
       );
       free(zComment);
     }
   }
   db_end_transaction(0);
-  manifest_clear(&m);
-  return 1;
-}
-
-/*
-** Given a checkin name, load and parse the manifest for that checkin.
-** Throw a fatal error if anything goes wrong.
-*/
-void manifest_from_name(
-  const char *zName,
-  Manifest *pM
-){
-  int rid;
-  Blob content;
-
-  rid = name_to_rid(zName);
-  if( !is_a_version(rid) ){
-    fossil_fatal("no such checkin: %s", zName);
-  }
-  content_get(rid, &content);
-  if( !manifest_parse(pM, &content) ){
-    fossil_fatal("cannot parse manifest for checkin: %s", zName);
-  }
+  if( p->type==CFTYPE_MANIFEST ){
+    manifest_cache_insert(p);
+  }else{
+    manifest_destroy(p);
+  }
+  return 1;
 }

Index: src/md5.c
==================================================================
--- src/md5.c
+++ src/md5.c
@@ -41,12 +41,16 @@
   uint32 bits[2];
   unsigned char in[64];
 };
 typedef struct Context MD5Context;
 
+#if defined(__i386__) || defined(__x86_64__) || defined(_WIN32)
+# define byteReverse(A,B)
+#else
 /*
- * Note: this code is harmless on little-endian machines.
+ * Convert an array of integers to little-endian.
+ * Note: this code is a no-op on little-endian machines.
  */
 static void byteReverse (unsigned char *buf, unsigned longs){
         uint32 t;
         do {
                 t = (uint32)((unsigned)buf[3]<<8 | buf[2]) << 16 |
@@ -53,10 +57,12 @@
                             ((unsigned)buf[1]<<8 | buf[0]);
                 *(uint32 *)buf = t;
                 buf += 4;
         } while (--longs);
 }
+#endif
+
 /* The four core functions - F1 is optimized somewhat */
 
 /* #define F1(x, y, z) (x & y | ~x & z) */
 #define F1(x, y, z) (z ^ (x & (y ^ z)))
 #define F2(x, y, z) F1(z, x, y)
@@ -314,10 +320,30 @@
 ** Add the content of a blob to the incremental MD5 checksum.
 */
 void md5sum_step_blob(Blob *p){
   md5sum_step_text(blob_buffer(p), blob_size(p));
 }
+
+/*
+** For trouble-shooting only:
+**
+** Report the current state of the incremental checksum.
+*/
+const char *md5sum_current_state(void){
+  unsigned int cksum = 0;
+  unsigned int *pFirst, *pLast;
+  static char zResult[12];
+
+  pFirst = (unsigned int*)&incrCtx;
+  pLast = (unsigned int*)((&incrCtx)+1);
+  while( pFirst pV1 (into aC1)
   ** and pPivot => pV2 (into aC2).  Each of the aC1 and aC2 arrays is
@@ -165,12 +165,12 @@
   ** is the number of lines of text to copy directly from the pivot,
   ** the second integer is the number of lines of text to omit from the
   ** pivot, and the third integer is the number of lines of text that are
   ** inserted.  The edit array ends with a triple of 0,0,0.
   */
-  aC1 = text_diff(pPivot, pV1, 0, 0);
-  aC2 = text_diff(pPivot, pV2, 0, 0);
+  aC1 = text_diff(pPivot, pV1, 0, 0, 1);
+  aC2 = text_diff(pPivot, pV2, 0, 0, 1);
   if( aC1==0 || aC2==0 ){
     free(aC1);
     free(aC2);
     return -1;
   }

Index: src/name.c
==================================================================
--- src/name.c
+++ src/name.c
@@ -111,20 +111,20 @@
 
 /*
 ** Return TRUE if the string begins with an ISO8601 date: YYYY-MM-DD.
 */
 static int is_date(const char *z){
-  if( !isdigit(z[0]) ) return 0;
-  if( !isdigit(z[1]) ) return 0;
-  if( !isdigit(z[2]) ) return 0;
-  if( !isdigit(z[3]) ) return 0;
+  if( !fossil_isdigit(z[0]) ) return 0;
+  if( !fossil_isdigit(z[1]) ) return 0;
+  if( !fossil_isdigit(z[2]) ) return 0;
+  if( !fossil_isdigit(z[3]) ) return 0;
   if( z[4]!='-') return 0;
-  if( !isdigit(z[5]) ) return 0;
-  if( !isdigit(z[6]) ) return 0;
+  if( !fossil_isdigit(z[5]) ) return 0;
+  if( !fossil_isdigit(z[6]) ) return 0;
   if( z[7]!='-') return 0;
-  if( !isdigit(z[8]) ) return 0;
-  if( !isdigit(z[9]) ) return 0;
+  if( !fossil_isdigit(z[8]) ) return 0;
+  if( !fossil_isdigit(z[9]) ) return 0;
   return 1;
 }
 
 /*
 ** Convert a symbolic tag name into the UUID of a check-in that contains
@@ -280,11 +280,11 @@
 
   if( zName==0 || zName[0]==0 ) return 0;
   blob_init(&name, zName, -1);
   if( name_to_uuid(&name, -1) ){
     blob_reset(&name);
-    for(i=0; zName[i] && isdigit(zName[i]); i++){}
+    for(i=0; zName[i] && fossil_isdigit(zName[i]); i++){}
     if( zName[i]==0 ){
       rid = atoi(zName);
       if( db_exists("SELECT 1 FROM blob WHERE rid=%d", rid) ){
         return rid;
       }
@@ -348,11 +348,11 @@
   if( zName==0 || zName[0]==0 ) return 0;
   blob_init(&name, zName, -1);
   rc = name_to_uuid(&name, -1);
   if( rc==1 ){
     blob_reset(&name);
-    for(i=0; zName[i] && isdigit(zName[i]); i++){}
+    for(i=0; zName[i] && fossil_isdigit(zName[i]); i++){}
     if( zName[i]==0 ){
       rid = atoi(zName);
       if( db_exists("SELECT 1 FROM blob WHERE rid=%d", rid) ){
         return rid;
       }

ADDED   src/popen.c
Index: src/popen.c
==================================================================
--- src/popen.c
+++ src/popen.c
@@ -0,0 +1,187 @@
+/*
+** Copyright (c) 2010 D. Richard Hipp
+**
+** This program is free software; you can redistribute it and/or
+** modify it under the terms of the Simplified BSD License (also
+** known as the "2-Clause License" or "FreeBSD License".)
+
+** This program is distributed in the hope that it will be useful,
+** but without any warranty; without even the implied warranty of
+** merchantability or fitness for a particular purpose.
+**
+** Author contact information:
+**   drh@hwaci.com
+**   http://www.hwaci.com/drh/
+**
+*******************************************************************************
+**
+** This file contains an implementation of a bi-directional popen().
+*/
+#include "config.h"
+#include "popen.h"
+
+#ifdef _WIN32
+#include 
+#include 
+/*
+** Print a fatal error and quit.
+*/
+static void win32_fatal_error(const char *zMsg){
+  fossil_fatal("%s");
+}
+#endif
+
+
+
+#ifdef _WIN32
+/*
+** On windows, create a child process and specify the stdin, stdout,
+** and stderr channels for that process to use.
+**
+** Return the number of errors.
+*/
+static int win32_create_child_process(
+  char *zCmd,          /* The command that the child process will run */
+  HANDLE hIn,          /* Standard input */
+  HANDLE hOut,         /* Standard output */
+  HANDLE hErr,         /* Standard error */
+  DWORD *pChildPid     /* OUT: Child process handle */
+){
+  STARTUPINFO si;
+  PROCESS_INFORMATION pi;
+  BOOL rc;
+
+  memset(&si, 0, sizeof(si));
+  si.cb = sizeof(si);
+  si.dwFlags = STARTF_USESTDHANDLES;
+  SetHandleInformation(hIn, HANDLE_FLAG_INHERIT, TRUE);
+  si.hStdInput  = hIn;
+  SetHandleInformation(hOut, HANDLE_FLAG_INHERIT, TRUE);
+  si.hStdOutput = hOut;
+  SetHandleInformation(hErr, HANDLE_FLAG_INHERIT, TRUE);
+  si.hStdError  = hErr;
+  rc = CreateProcess(
+     NULL,  /* Application Name */
+     zCmd,  /* Command-line */
+     NULL,  /* Process attributes */
+     NULL,  /* Thread attributes */
+     TRUE,  /* Inherit Handles */
+     0,     /* Create flags  */
+     NULL,  /* Environment */
+     NULL,  /* Current directory */
+     &si,   /* Startup Info */
+     &pi    /* Process Info */
+  );
+  if( rc ){
+    CloseHandle( pi.hProcess );
+    CloseHandle( pi.hThread );
+    *pChildPid = pi.dwProcessId;
+  }else{
+    win32_fatal_error("cannot create child process");
+  }
+  return rc!=0;
+}
+#endif
+
+/*
+** Create a child process running shell command "zCmd".  *ppOut is
+** a FILE that becomes the standard input of the child process.  
+** (The caller writes to *ppOut in order to send text to the child.)
+** *ppIn is stdout from the child process.  (The caller
+** reads from *ppIn in order to receive input from the child.)
+** Note that *ppIn is an unbuffered file descriptor, not a FILE.
+** The process ID of the child is written into *pChildPid.
+**
+** Return the number of errors.
+*/
+int popen2(const char *zCmd, int *pfdIn, FILE **ppOut, int *pChildPid){
+#ifdef _WIN32
+  HANDLE hStdinRd, hStdinWr, hStdoutRd, hStdoutWr, hStderr;
+  SECURITY_ATTRIBUTES saAttr;    
+  DWORD childPid = 0;
+  int fd;
+
+  saAttr.nLength = sizeof(saAttr);
+  saAttr.bInheritHandle = TRUE;
+  saAttr.lpSecurityDescriptor = NULL; 
+  hStderr = GetStdHandle(STD_ERROR_HANDLE);
+  if( !CreatePipe(&hStdoutRd, &hStdoutWr, &saAttr, 4096) ){
+    win32_fatal_error("cannot create pipe for stdout");
+  }
+  SetHandleInformation( hStdoutRd, HANDLE_FLAG_INHERIT, FALSE);
+
+  if( !CreatePipe(&hStdinRd, &hStdinWr, &saAttr, 4096) ){
+    win32_fatal_error("cannot create pipe for stdin");
+  }
+  SetHandleInformation( hStdinWr, HANDLE_FLAG_INHERIT, FALSE);
+  
+  win32_create_child_process((char*)zCmd, 
+                             hStdinRd, hStdoutWr, hStderr,&childPid);
+  *pChildPid = childPid;
+  *pfdIn = _open_osfhandle((long)hStdoutRd, 0);
+  fd = _open_osfhandle((long)hStdinWr, 0);
+  *ppOut = _fdopen(fd, "w");
+  CloseHandle(hStdinRd); 
+  CloseHandle(hStdoutWr);
+  return 0;
+#else
+  int pin[2], pout[2];
+  *pfdIn = 0;
+  *ppOut = 0;
+  *pChildPid = 0;
+
+  if( pipe(pin)<0 ){
+    return 1;
+  }
+  if( pipe(pout)<0 ){
+    close(pin[0]);
+    close(pin[1]);
+    return 1;
+  }
+  *pChildPid = fork();
+  if( *pChildPid<0 ){
+    close(pin[0]);
+    close(pin[1]);
+    close(pout[0]);
+    close(pout[1]);
+    *pChildPid = 0;
+    return 1;
+  }
+  if( *pChildPid==0 ){
+    /* This is the child process */
+    close(0);
+    dup(pout[0]);
+    close(pout[0]);
+    close(pout[1]);
+    close(1);
+    dup(pin[1]);
+    close(pin[0]);
+    close(pin[1]);
+    execl("/bin/sh", "/bin/sh", "-c", zCmd, (char*)0);
+    return 1;
+  }else{
+    /* This is the parent process */
+    close(pin[1]);
+    *pfdIn = pin[0];
+    close(pout[0]);
+    *ppOut = fdopen(pout[1], "w");
+    return 0;
+  }
+#endif
+}
+
+/*
+** Close the connection to a child process previously created using
+** popen2().  Kill off the child process, then close the pipes.
+*/
+void pclose2(int fdIn, FILE *pOut, int childPid){
+#ifdef _WIN32
+  /* Not implemented, yet */
+  close(fdIn);
+  fclose(pOut);
+#else
+  close(fdIn);
+  fclose(pOut);
+  kill(childPid, SIGINT);
+#endif
+}

Index: src/pqueue.c
==================================================================
--- src/pqueue.c
+++ src/pqueue.c
@@ -62,11 +62,11 @@
 
 /*
 ** Change the size of the queue so that it contains N slots
 */
 static void pqueue_resize(PQueue *p, int N){
-  p->a = realloc(p->a, sizeof(p->a[0])*N);
+  p->a = fossil_realloc(p->a, sizeof(p->a[0])*N);
   p->sz = N;
 }
 
 /*
 ** Insert element e into the queue.

Index: src/printf.c
==================================================================
--- src/printf.c
+++ src/printf.c
@@ -44,13 +44,13 @@
 #define etHTMLIZE    16 /* Make text safe for HTML */
 #define etHTTPIZE    17 /* Make text safe for HTTP.  "/" encoded as %2f */
 #define etURLIZE     18 /* Make text safe for HTTP.  "/" not encoded */
 #define etFOSSILIZE  19 /* The fossil header encoding format. */
 #define etPATH       20 /* Path type */
-#define etWIKISTR    21 /* Wiki text rendered from a char* */
-#define etWIKIBLOB   22 /* Wiki text rendered from a Blob* */
-#define etSTRINGID   23 /* String with length limit for a UUID prefix */
+#define etWIKISTR    21 /* Wiki text rendered from a char*: %w */
+#define etWIKIBLOB   22 /* Wiki text rendered from a Blob*: %W */
+#define etSTRINGID   23 /* String with length limit for a UUID prefix: %S */
 
 
 /*
 ** An "etByte" is an 8-bit unsigned value.
 */
@@ -560,11 +560,11 @@
         int i;
         int limit = flag_alternateform ? va_arg(ap,int) : -1;
         char *e = va_arg(ap,char*);
         if( e==0 ){e="";}
         length = StrNLen32(e, limit);
-        zExtra = bufpt = malloc(length+1);
+        zExtra = bufpt = fossil_malloc(length+1);
         for( i=0; i=0 && limit etBUFSIZE ){
-          bufpt = zExtra = malloc( n + cnt + 2 );
+          bufpt = zExtra = fossil_malloc( n + cnt + 2 );
         }else{
           bufpt = buf;
         }
         bufpt[0] = '\'';
         for(i=0, j=1; ietBUFSIZE ){
-          bufpt = zExtra = malloc( n );
-          if( bufpt==0 ) return -1;
+          bufpt = zExtra = fossil_malloc( n );
         }else{
           bufpt = buf;
         }
         j = 0;
         if( needQuote ) bufpt[j++] = '\'';

Index: src/rebuild.c
==================================================================
--- src/rebuild.c
+++ src/rebuild.c
@@ -72,100 +72,146 @@
 @   content TEXT
 @ );
 ;
 
 /*
-** Variables used for progress information
+** Variables used to store state information about an on-going "rebuild"
+** or "deconstruct".
 */
 static int totalSize;       /* Total number of artifacts to process */
 static int processCnt;      /* Number processed so far */
 static int ttyOutput;       /* Do progress output */
 static Bag bagDone;         /* Bag of records rebuilt */
+
+static char *zFNameFormat;  /* Format string for filenames on deconstruct */
+static int prefixLength;    /* Length of directory prefix for deconstruct */
+
+
+/*
+** Draw the percent-complete message.
+** The input is actually the permill complete.
+*/
+static void percent_complete(int permill){
+  static int lastOutput = -1;
+  if( permill>lastOutput ){
+    printf("  %d.%d%% complete...\r", permill/10, permill%10);
+    fflush(stdout);
+    lastOutput = permill;
+  }
+}
+
 
 /*
 ** Called after each artifact is processed
 */
 static void rebuild_step_done(rid){
   /* assert( bag_find(&bagDone, rid)==0 ); */
   bag_insert(&bagDone, rid);
   if( ttyOutput ){
     processCnt++;
-    if (!g.fQuiet) {
-      printf("%d (%d%%)...\r", processCnt, (processCnt*100/totalSize));
-      fflush(stdout);
+    if (!g.fQuiet && totalSize>0) {
+      percent_complete((processCnt*1000)/totalSize);
     }
   }
 }
 
 /*
 ** Rebuild cross-referencing information for the artifact
 ** rid with content pBase and all of its descendants.  This
 ** routine clears the content buffer before returning.
+**
+** If the zFNameFormat variable is set, then this routine is
+** called to run "fossil deconstruct" instead of the usual
+** "fossil rebuild".  In that case, instead of rebuilding the
+** cross-referencing information, write the file content out
+** to the approriate directory.
+**
+** In both cases, this routine automatically recurses to process
+** other artifacts that are deltas off of the current artifact.
+** This is the most efficient way to extract all of the original
+** artifact content from the Fossil repository.
 */
 static void rebuild_step(int rid, int size, Blob *pBase){
   static Stmt q1;
   Bag children;
   Blob copy;
   Blob *pUse;
   int nChild, i, cid;
 
-  /* Fix up the "blob.size" field if needed. */
-  if( size!=blob_size(pBase) ){
-    db_multi_exec(
-       "UPDATE blob SET size=%d WHERE rid=%d", blob_size(pBase), rid
-    );
-  }
-
-  /* Find all children of artifact rid */
-  db_static_prepare(&q1, "SELECT rid FROM delta WHERE srcid=:rid");
-  db_bind_int(&q1, ":rid", rid);
-  bag_init(&children);
-  while( db_step(&q1)==SQLITE_ROW ){
-    int cid = db_column_int(&q1, 0);
-    if( !bag_find(&bagDone, cid) ){
-      bag_insert(&children, cid);
-    }
-  }
-  nChild = bag_count(&children);
-  db_reset(&q1);
-
-  /* Crosslink the artifact */
-  if( nChild==0 ){
-    pUse = pBase;
-  }else{
-    blob_copy(©, pBase);
-    pUse = ©
-  }
-  manifest_crosslink(rid, pUse);
-  blob_reset(pUse);
-
-  /* Call all children recursively */
-  for(cid=bag_first(&children), i=1; cid; cid=bag_next(&children, cid), i++){
-    Stmt q2;
-    int sz;
-    if( nChild==i ){
-      pUse = pBase;
-    }else{
-      blob_copy(©, pBase);
-      pUse = ©
-    }
-    db_prepare(&q2, "SELECT content, size FROM blob WHERE rid=%d", cid);
-    if( db_step(&q2)==SQLITE_ROW && (sz = db_column_int(&q2,1))>=0 ){
-      Blob delta;
-      db_ephemeral_blob(&q2, 0, &delta);
-      blob_uncompress(&delta, &delta);
-      blob_delta_apply(pUse, &delta, pUse);
-      blob_reset(&delta);
-      db_finalize(&q2);
-      rebuild_step(cid, sz, pUse);
-    }else{
-      db_finalize(&q2);
-      blob_reset(pUse);
-    }
-  }
-  bag_clear(&children);
-  rebuild_step_done(rid);
+  while( rid>0 ){
+
+    /* Fix up the "blob.size" field if needed. */
+    if( size!=blob_size(pBase) ){
+      db_multi_exec(
+         "UPDATE blob SET size=%d WHERE rid=%d", blob_size(pBase), rid
+      );
+    }
+  
+    /* Find all children of artifact rid */
+    db_static_prepare(&q1, "SELECT rid FROM delta WHERE srcid=:rid");
+    db_bind_int(&q1, ":rid", rid);
+    bag_init(&children);
+    while( db_step(&q1)==SQLITE_ROW ){
+      int cid = db_column_int(&q1, 0);
+      if( !bag_find(&bagDone, cid) ){
+        bag_insert(&children, cid);
+      }
+    }
+    nChild = bag_count(&children);
+    db_reset(&q1);
+  
+    /* Crosslink the artifact */
+    if( nChild==0 ){
+      pUse = pBase;
+    }else{
+      blob_copy(©, pBase);
+      pUse = ©
+    }
+    if( zFNameFormat==0 ){
+      /* We are doing "fossil rebuild" */
+      manifest_crosslink(rid, pUse);
+    }else{
+      /* We are doing "fossil deconstruct" */
+      char *zUuid = db_text(0, "SELECT uuid FROM blob WHERE rid=%d", rid);
+      char *zFile = mprintf(zFNameFormat, zUuid, zUuid+prefixLength);
+      blob_write_to_file(pUse,zFile);
+      free(zFile);
+      free(zUuid);
+    }
+    blob_reset(pUse);
+    rebuild_step_done(rid);
+  
+    /* Call all children recursively */
+    rid = 0;
+    for(cid=bag_first(&children), i=1; cid; cid=bag_next(&children, cid), i++){
+      static Stmt q2;
+      int sz;
+      db_static_prepare(&q2, "SELECT content, size FROM blob WHERE rid=:rid");
+      db_bind_int(&q2, ":rid", cid);
+      if( db_step(&q2)==SQLITE_ROW && (sz = db_column_int(&q2,1))>=0 ){
+        Blob delta, next;
+        db_ephemeral_blob(&q2, 0, &delta);
+        blob_uncompress(&delta, &delta);
+        blob_delta_apply(pBase, &delta, &next);
+        blob_reset(&delta);
+        db_reset(&q2);
+        if( i0 ){
+    processCnt += incrSize;
+    percent_complete((processCnt*1000)/totalSize);
+  }
+  create_cluster();
+  if( !g.fQuiet && totalSize>0 ){
+    processCnt += incrSize;
+    percent_complete((processCnt*1000)/totalSize);
+  }
   if(!g.fQuiet && ttyOutput ){
     printf("\n");
   }
   return errCnt;
 }
@@ -295,11 +352,13 @@
 */
 void rebuild_database(void){
   int forceFlag;
   int randomizeFlag;
   int errCnt;
+  int omitVerify;
 
+  omitVerify = find_option("noverify",0,0)!=0;
   forceFlag = find_option("force","f",0)!=0;
   randomizeFlag = find_option("randomize", 0, 0)!=0;
   if( g.argc==3 ){
     db_open_repository(g.argv[2]);
   }else{
@@ -316,10 +375,11 @@
   if( errCnt && !forceFlag ){
     printf("%d errors. Rolling back changes. Use --force to force a commit.\n",
             errCnt);
     db_end_transaction(1);
   }else{
+    if( omitVerify ) verify_cancel();
     db_end_transaction(0);
   }
 }
 
 /*
@@ -340,10 +400,32 @@
     "UPDATE config SET value='detached-' || value"
     " WHERE name='project-name' AND value NOT GLOB 'detached-*';"
   );
   db_end_transaction(0);
 }
+
+/*
+** COMMAND:  test-create-clusters
+**
+** Create clusters for all unclustered artifacts if the number of unclustered
+** artifacts exceeds the current clustering threshold.
+*/
+void test_createcluster_cmd(void){
+  if( g.argc==3 ){
+    db_open_repository(g.argv[2]);
+  }else{
+    db_find_and_open_repository(1);
+    if( g.argc!=2 ){
+      usage("?REPOSITORY-FILENAME?");
+    }
+    db_close();
+    db_open_repository(g.zRepositoryName);
+  }
+  db_begin_transaction();
+  create_cluster();
+  db_end_transaction(0);  
+}
 
 /*
 ** COMMAND: scrub
 ** %fossil scrub [--verily] [--force] [REPOSITORY]
 **
@@ -401,26 +483,66 @@
   }else{
     rebuild_db(0, 1);
     db_end_transaction(0);
   }
 }
+
+/*
+** Recursively read all files from the directory zPath and install
+** every file read as a new artifact in the repository.
+*/
+void recon_read_dir(char *zPath){
+  DIR *d;
+  struct dirent *pEntry;
+  Blob aContent; /* content of the just read artifact */
+  static int nFileRead = 0;
+
+  d = opendir(zPath);
+  if( d ){
+    while( (pEntry=readdir(d))!=0 ){
+      Blob path;
+      char *zSubpath;
+
+      if( pEntry->d_name[0]=='.' ){
+        continue;
+      }
+      zSubpath = mprintf("%s/%s",zPath,pEntry->d_name);
+      if( file_isdir(zSubpath)==1 ){
+        recon_read_dir(zSubpath);
+      }
+      blob_init(&path, 0, 0);
+      blob_appendf(&path, "%s", zSubpath);
+      if( blob_read_from_file(&aContent, blob_str(&path))==-1 ){
+        fossil_panic("some unknown error occurred while reading \"%s\"", 
+                     blob_str(&path));
+      }
+      content_put(&aContent, 0, 0);
+      blob_reset(&path);
+      blob_reset(&aContent);
+      free(zSubpath);
+      printf("\r%d", ++nFileRead);
+      fflush(stdout);
+    }
+  }else {
+    fossil_panic("encountered error %d while trying to open \"%s\".",
+                  errno, g.argv[3]);
+  }
+}
 
 /*
 ** COMMAND: reconstruct
 **
 ** Usage: %fossil reconstruct FILENAME DIRECTORY
 **
 ** This command studies the artifacts (files) in DIRECTORY and
 ** reconstructs the fossil record from them. It places the new
-** fossil repository in FILENAME
+** fossil repository in FILENAME. Subdirectories are read, files
+** with leading '.' in the filename are ignored.
 **
 */
 void reconstruct_cmd(void) {
   char *zPassword;
-  DIR *d;
-  struct dirent *pEntry;
-  Blob aContent; /* content of the just read artifact */
   if( g.argc!=4 ){
     usage("FILENAME DIRECTORY");
   }
   if( file_isdir(g.argv[3])!=1 ){
     printf("\"%s\" is not a directory\n\n", g.argv[3]);
@@ -430,35 +552,124 @@
   db_open_repository(g.argv[2]);
   db_open_config(0);
   db_begin_transaction();
   db_initial_setup(0, 0, 1);
 
-  d = opendir(g.argv[3]);
-  if( d ){
-    while( (pEntry=readdir(d))!=0 ){
-      Blob path;
-      blob_init(&path, 0, 0);
-      if( pEntry->d_name[0]=='.' ){
-        continue;
-      }
-      if( file_isdir(pEntry->d_name)==1 ){
-        continue;
-      }
-      blob_appendf(&path, "%s/%s", g.argv[3], pEntry->d_name);
-      if( blob_read_from_file(&aContent, blob_str(&path))==-1 ){
-        fossil_panic("Some unknown error occurred while reading \"%s\"", blob_str(&path));
-      }
-      content_put(&aContent, 0, 0);
-    }
-  }
-  else {
-    fossil_panic("Encountered error %d while trying to open \"%s\".", errno, g.argv[3]);
-  }
+  printf("Reading files from directory \"%s\"...\n", g.argv[3]);
+  recon_read_dir(g.argv[3]);
+  printf("\nBuilding the Fossil repository...\n");
 
   rebuild_db(0, 1);
 
+  /* Skip the verify_before_commit() step on a reconstruct.  Most artifacts
+  ** will have been changed and verification therefore takes a really, really
+  ** long time.
+  */
+  verify_cancel();
+  
   db_end_transaction(0);
   printf("project-id: %s\n", db_get("project-code", 0));
   printf("server-id: %s\n", db_get("server-code", 0));
   zPassword = db_text(0, "SELECT pw FROM user WHERE login=%Q", g.zLogin);
   printf("admin-user: %s (initial password is \"%s\")\n", g.zLogin, zPassword);
 }
+
+/*
+** COMMAND: deconstruct
+**
+** Usage %fossil deconstruct ?-R|--repository REPOSITORY? ?-L|--prefixlength N? DESTINATION
+**
+** This command exports all artifacts of o given repository and
+** writes all artifacts to the file system. The DESTINATION directory
+** will be populated with subdirectories AA and files AA/BBBBBBBBB.., where
+** AABBBBBBBBB.. is the 40 character artifact ID, AA the first 2 characters.
+** If -L|--prefixlength is given, the length (default 2) of the directory
+** prefix can be set to 0,1,..,9 characters.
+*/
+void deconstruct_cmd(void){
+  const char *zDestDir;
+  const char *zPrefixOpt;
+  Stmt        s;
+
+  /* check number of arguments */
+  if( (g.argc != 3) && (g.argc != 5)  && (g.argc != 7)){
+    usage ("?-R|--repository REPOSITORY? ?-L|--prefixlength N? DESTINATION");
+  }
+  /* get and check argument destination directory */
+  zDestDir = g.argv[g.argc-1];
+  if( !*zDestDir  || !file_isdir(zDestDir)) {
+    fossil_panic("DESTINATION(%s) is not a directory!",zDestDir);
+  }
+  /* get and check prefix length argument and build format string */
+  zPrefixOpt=find_option("prefixlength","L",1);
+  if( !zPrefixOpt ){
+    prefixLength = 2;
+  }else{
+    if( zPrefixOpt[0]>='0' && zPrefixOpt[0]<='9' && !zPrefixOpt[1] ){
+      prefixLength = (int)(*zPrefixOpt-'0');
+    }else{
+      fossil_fatal("N(%s) is not a a valid prefix length!",zPrefixOpt);
+    }
+  }
+#ifndef _WIN32
+  if( access(zDestDir, W_OK) ){
+    fossil_fatal("DESTINATION(%s) is not writeable!",zDestDir);
+  }
+#else
+  /* write access on windows is not checked, errors will be
+  ** dected on blob_write_to_file
+  */
+#endif
+  if( prefixLength ){
+    zFNameFormat = mprintf("%s/%%.%ds/%%s",zDestDir,prefixLength);
+  }else{
+    zFNameFormat = mprintf("%s/%%s",zDestDir);
+  }
+  /* open repository and open query for all artifacts */
+  db_find_and_open_repository(1);
+  bag_init(&bagDone);
+  ttyOutput = 1;
+  processCnt = 0;
+  if (!g.fQuiet) {
+    printf("0 (0%%)...\r");
+    fflush(stdout);
+  }
+  totalSize = db_int(0, "SELECT count(*) FROM blob");
+  db_prepare(&s,
+     "SELECT rid, size FROM blob /*scan*/"
+     " WHERE NOT EXISTS(SELECT 1 FROM shun WHERE uuid=blob.uuid)"
+     "   AND NOT EXISTS(SELECT 1 FROM delta WHERE rid=blob.rid)"
+  );
+  while( db_step(&s)==SQLITE_ROW ){
+    int rid = db_column_int(&s, 0);
+    int size = db_column_int(&s, 1);
+    if( size>=0 ){
+      Blob content;
+      content_get(rid, &content);
+      rebuild_step(rid, size, &content);
+    }
+  }
+  db_finalize(&s);
+  db_prepare(&s,
+     "SELECT rid, size FROM blob"
+     " WHERE NOT EXISTS(SELECT 1 FROM shun WHERE uuid=blob.uuid)"
+  );
+  while( db_step(&s)==SQLITE_ROW ){
+    int rid = db_column_int(&s, 0);
+    int size = db_column_int(&s, 1);
+    if( size>=0 ){
+      if( !bag_find(&bagDone, rid) ){
+        Blob content;
+        content_get(rid, &content);
+        rebuild_step(rid, size, &content);
+      }
+    }
+  }
+  db_finalize(&s);
+  if(!g.fQuiet && ttyOutput ){
+    printf("\n");
+  }
+
+  /* free filename format string */
+  free(zFNameFormat);
+  zFNameFormat = 0;
+}

Index: src/report.c
==================================================================
--- src/report.c
+++ src/report.c
@@ -88,22 +88,22 @@
 /*
 ** Remove whitespace from both ends of a string.
 */
 char *trim_string(const char *zOrig){
   int i;
-  while( isspace(*zOrig) ){ zOrig++; }
+  while( fossil_isspace(*zOrig) ){ zOrig++; }
   i = strlen(zOrig);
-  while( i>0 && isspace(zOrig[i-1]) ){ i--; }
+  while( i>0 && fossil_isspace(zOrig[i-1]) ){ i--; }
   return mprintf("%.*s", i, zOrig);
 }
 
 /*
 ** Extract a numeric (integer) value from a string.
 */
 char *extract_integer(const char *zOrig){
   if( zOrig == NULL || zOrig[0] == 0 ) return "";
-  while( *zOrig && !isdigit(*zOrig) ){ zOrig++; }
+  while( *zOrig && !fossil_isdigit(*zOrig) ){ zOrig++; }
   if( *zOrig ){
     /* we have a digit. atoi() will get as much of the number as it
     ** can. We'll run it through mprintf() to get a string. Not
     ** an efficient way to do it, but effective.
     */
@@ -118,18 +118,18 @@
 ** which also converts any CRNL sequence into a single NL.
 */
 char *remove_blank_lines(const char *zOrig){
   int i, j, n;
   char *z;
-  for(i=j=0; isspace(zOrig[i]); i++){ if( zOrig[i]=='\n' ) j = i+1; }
+  for(i=j=0; fossil_isspace(zOrig[i]); i++){ if( zOrig[i]=='\n' ) j = i+1; }
   n = strlen(&zOrig[j]);
-  while( n>0 && isspace(zOrig[j+n-1]) ){ n--; }
+  while( n>0 && fossil_isspace(zOrig[j+n-1]) ){ n--; }
   z = mprintf("%.*s", n, &zOrig[j]);
   for(i=j=0; z[i]; i++){
-    if( z[i+1]=='\n' && z[i]!='\n' && isspace(z[i]) ){
+    if( z[i+1]=='\n' && z[i]!='\n' && fossil_isspace(z[i]) ){
       z[j] = z[i];
-      while(isspace(z[j]) && z[j] != '\n' ){ j--; }
+      while(fossil_isspace(z[j]) && z[j] != '\n' ){ j--; }
       j++;
       continue;
     }
 
     z[j++] = z[i];
@@ -145,11 +145,11 @@
 ** This is the SQLite authorizer callback used to make sure that the
 ** SQL statements entered by users do not try to do anything untoward.
 ** If anything suspicious is tried, set *(char**)pError to an error
 ** message obtained from malloc.
 */
-static int report_query_authorizer(
+int report_query_authorizer(
   void *pError,
   int code,
   const char *zArg1,
   const char *zArg2,
   const char *zArg3,
@@ -212,11 +212,11 @@
   int rc;
 
   /* First make sure the SQL is a single query command by verifying that
   ** the first token is "SELECT" and that there are no unquoted semicolons.
   */
-  for(i=0; isspace(zSql[i]); i++){}
+  for(i=0; fossil_isspace(zSql[i]); i++){}
   if( strncasecmp(&zSql[i],"select",6)!=0 ){
     return mprintf("The SQL must be a SELECT statement");
   }
   for(i=0; zSql[i]; i++){
     if( zSql[i]==';' ){
@@ -277,13 +277,13 @@
   zSQL = db_column_text(&q, 1);
   zOwner = db_column_text(&q, 2);
   zClrKey = db_column_text(&q, 3);
   @ 
   @ 
-  @ 
+  @ 
   @ 
-  @ 
+  @ 
   @ 
   @ 
   @ 
-  }else{
-    if(set){
-      @ 
-  }
-}
-#endif
-
 /*
 ** The state of the report generation.
 */
 struct GenerateHTML {
   int rn;          /* Report number */
@@ -723,11 +694,11 @@
 
   /* Output the data for this entry from the database
   */
   zBg = pState->iBg>=0 ? azArg[pState->iBg] : 0;
   if( zBg==0 ) zBg = "white";
-  @ 
+  @ 
   zTid = 0;
   zPage[0] = 0;
   for(i=0; iiBg ) continue;
@@ -738,11 +709,11 @@
         @ 
         zTid = 0;
       }
       if( zData[0] ){
         Blob content;
-        @ Title: %h(zTitle)
%h(zTitle)
Owner: %h(zOwner)
%h(zOwner)
SQL:    @ %h(zSQL)
   @ 
@@ -328,11 +328,11 @@
     zTitle = db_text(0, "SELECT title FROM reportfmt "
                          "WHERE rn=%d", rn);
     if( zTitle==0 ) cgi_redirect("reportlist");
 
     style_header("Are You Sure?");
-    @ 
+    @ 
     @ You are about to delete all traces of the report
     @ %h(zTitle) from
     @ the database.  This is an irreversible operation.  All records
     @ related to this report will be removed and cannot be recovered.
     @
@@ -393,35 +393,35 @@
     }
   }
   if( zOwner==0 ) zOwner = g.zLogin;
   style_submenu_element("Cancel", "Cancel", "reportlist");
   if( rn>0 ){
-    style_submenu_element("Delete", "Delete", "rptedit?rn=%d&del1=1", rn);
+    style_submenu_element("Delete", "Delete", "rptedit?rn=%d&del1=1", rn);
   }
   style_header(rn>0 ? "Edit Report Format":"Create New Report Format");
   if( zErr ){
-    @ %h(zErr)
+    @ %h(zErr)
   }
-  @ 
-  @ 
-  @ Report Title:

-  @ 
-  @ Enter a complete SQL query statement against the "TICKET" table:

+  @ 

+  @ 
+  @ Report Title:

+  @ 
+  @ Enter a complete SQL query statement against the "TICKET" table:

   @ %h(zSQL)
   @ 
   login_insert_csrf_secret();
   if( g.okAdmin ){
     @ Report owner:
-    @ 
+    @ 
     @ 
   } else {
-    @ 
+    @ 
   }
   @ Enter an optional color key in the following box.  (If blank, no
   @ color key is displayed.)  Each line contains the text for a single
   @ entry in the key.  The first token of each line is the background
-  @ color for that line.

+  @ color for that line.

   @ %h(zClrKey)
   @ 
   if( !g.okAdmin && strcmp(zOwner,g.zLogin)!=0 ){
     @ This report format is owned by %h(zOwner).  You are not allowed
     @ to change it.
@@ -428,15 +428,15 @@
     @ 
     report_format_hints();
     style_footer();
     return;
   }
-  @ 
+  @ 
   if( rn>0 ){
-    @ 
+    @ 
   }
-  @ 
+  @ 
   report_format_hints();
   style_footer();
 }
 
 /*
@@ -448,11 +448,11 @@
   zSchema = db_text(0,"SELECT sql FROM sqlite_master WHERE name='ticket'");
   if( zSchema==0 ){
     zSchema = db_text(0,"SELECT sql FROM repository.sqlite_master"
                         " WHERE name='ticket'");
   }
-  @ 
TICKET Schema
+  @ 
TICKET Schema
   @    @ %h(zSchema)
   @ 
   @ Notes
   @ 
@@ -478,17 +478,17 @@
   @ In this example, the first column in the result set is named
   @ "bgcolor".  The value of this column is not displayed.  Instead, it
   @ selects the background color of each row based on the TICKET.STATUS
   @ field of the database.  The color key at the right shows the various
   @ color codes.
-  @ 
-  @ 
-  @ 
-  @ 
-  @ 
-  @ 
-  @ 
+  @ new or active
review
fixed
tested
defer
closed
+  @ 
+  @ 
+  @ 
+  @ 
+  @ 
+  @ 
   @ new or active
review
fixed
tested
defer
closed
   @    @ SELECT
   @   CASE WHEN status IN ('new','active') THEN '#f2dcdc'
   @        WHEN status='review' THEN '#e8e8bd'
@@ -510,16 +510,16 @@
   @ FROM ticket
   @ 
   @ To base the background color on the TICKET.PRIORITY or
   @ TICKET.SEVERITY fields, substitute the following code for the
   @ first column of the query:
-  @ 
-  @ 
-  @ 
-  @ 
-  @ 
-  @ 
+  @ 1
2
3
4
5
+  @ 
+  @ 
+  @ 
+  @ 
+  @ 
   @ 1
2
3
4
5
   @    @ SELECT
   @   CASE priority WHEN 1 THEN '#f2dcdc'
   @        WHEN 2 THEN '#e8e8bd'
@@ -583,39 +583,10 @@
   @  FROM ticket
   @ 
   @
 }
 
-#if 0 /* NOT USED */
-static void column_header(int rn,const char *zCol, int nCol, int nSorted,
-    const char *zDirection, const char *zExtra
-){
-  int set = (nCol==nSorted);
-  int desc = !strcmp(zDirection,"DESC");
-
-  /*
-  ** Clicking same column header 3 times in a row resets any sorting.
-  ** Note that we link to rptview, which means embedded reports will get
-  ** sent to the actual report view page as soon as a user tries to do
-  ** any sorting. I don't see that as a Bad Thing.
-  */
-  if(set && desc){
-    @ 
-    @   %h(zCol) %h(zCol)
edit
nCol)>
+        @ 
nCol)>
         blob_init(&content, zData, -1);
         wiki_convert(&content, 0, 0);
         blob_reset(&content);
       }
     }else if( azName[i][0]=='#' ){
@@ -772,15 +743,15 @@
 ** spaces.
 */
 static void output_no_tabs(const char *z){
   while( z && z[0] ){
     int i, j;
-    for(i=0; z[i] && (!isspace(z[i]) || z[i]==' '); i++){}
+    for(i=0; z[i] && (!fossil_isspace(z[i]) || z[i]==' '); i++){}
     if( i>0 ){
       cgi_printf("%.*s", i, z);
     }
-    for(j=i; isspace(z[j]); j++){}
+    for(j=i; fossil_isspace(z[j]); j++){}
     if( j>i ){
       cgi_printf("%*s", j-i, "");
     }
     z += j;
   }
@@ -816,27 +787,27 @@
 ** Generate HTML that describes a color key.
 */
 void output_color_key(const char *zClrKey, int horiz, char *zTabArgs){
   int i, j, k;
   char *zSafeKey, *zToFree;
-  while( isspace(*zClrKey) ) zClrKey++;
+  while( fossil_isspace(*zClrKey) ) zClrKey++;
   if( zClrKey[0]==0 ) return;
   @ 
   if( horiz ){
     @ 
   }
   zToFree = zSafeKey = mprintf("%h", zClrKey);
   while( zSafeKey[0] ){
-    while( isspace(*zSafeKey) ) zSafeKey++;
-    for(i=0; zSafeKey[i] && !isspace(zSafeKey[i]); i++){}
-    for(j=i; isspace(zSafeKey[j]); j++){}
+    while( fossil_isspace(*zSafeKey) ) zSafeKey++;
+    for(i=0; zSafeKey[i] && !fossil_isspace(zSafeKey[i]); i++){}
+    for(j=i; fossil_isspace(zSafeKey[j]); j++){}
     for(k=j; zSafeKey[k] && zSafeKey[k]!='\n' && zSafeKey[k]!='\r'; k++){}
     if( !horiz ){
-      cgi_printf("\n",
+      cgi_printf("\n",
         i, zSafeKey, k-j, &zSafeKey[j]);
     }else{
-      cgi_printf("\n",
+      cgi_printf("\n",
         i, zSafeKey, k-j, &zSafeKey[j]);
     }
     zSafeKey += k;
   }
   free(zToFree);
@@ -907,11 +878,11 @@
   if( !tabs ){
     struct GenerateHTML sState;
 
     db_multi_exec("PRAGMA empty_result_callbacks=ON");
     style_submenu_element("Raw", "Raw", 
-      "rptview?tablist=1&%s", PD("QUERY_STRING",""));
+      "rptview?tablist=1&%h", PD("QUERY_STRING",""));
     if( g.okAdmin 
        || (g.okTktFmt && g.zLogin && zOwner && strcmp(g.zLogin,zOwner)==0) ){
       style_submenu_element("Edit", "Edit", "rptedit?rn=%d", rn);
     }
     if( g.okTktFmt ){
@@ -921,26 +892,189 @@
       style_submenu_element("New Ticket", "Create a new ticket",
         "%s/tktnew", g.zTop);
     }
     style_header(zTitle);
     output_color_key(zClrKey, 1, 
-        "border=0 cellpadding=3 cellspacing=0 class=\"report\"");
-    @ 
%.*s
%.*s
%.*s %.*s
+        "border=\"0\" cellpadding=\"3\" cellspacing=\"0\" class=\"report\"");
+    @ 
     sState.rn = rn;
     sState.nCount = 0;
     sqlite3_set_authorizer(g.db, report_query_authorizer, (void*)&zErr1);
     sqlite3_exec(g.db, zSql, generate_html, &sState, &zErr2);
     sqlite3_set_authorizer(g.db, 0, 0);
     @ 
     if( zErr1 ){
-      @ Error: %h(zErr1)
+      @ Error: %h(zErr1)
     }else if( zErr2 ){
-      @ Error: %h(zErr2)
+      @ Error: %h(zErr2)
     }
     style_footer();
   }else{
     sqlite3_set_authorizer(g.db, report_query_authorizer, (void*)&zErr1);
     sqlite3_exec(g.db, zSql, output_tab_separated, &count, &zErr2);
     sqlite3_set_authorizer(g.db, 0, 0);
     cgi_set_content_type("text/plain");
   }
 }
+
+/*
+** report number for full table ticket export
+*/
+static const char zFullTicketRptRn[] = "0";
+
+/*
+** report title for full table ticket export
+*/
+static const char zFullTicketRptTitle[] = "full ticket export";
+
+/*
+** show all reports, which can be used for ticket show.
+** Output is written to stdout as tab delimited table
+*/
+void rpt_list_reports(void){
+  Stmt q;
+  char const aRptOutFrmt[] = "%s\t%s\n";
+
+  printf("Available reports:\n");
+  printf(aRptOutFrmt,"report number","report title");
+  printf(aRptOutFrmt,zFullTicketRptRn,zFullTicketRptTitle);
+  db_prepare(&q,"SELECT rn,title FROM reportfmt ORDER BY rn");
+  while( db_step(&q)==SQLITE_ROW ){
+    const char *zRn = db_column_text(&q, 0);
+    const char *zTitle = db_column_text(&q, 1);
+
+    printf(aRptOutFrmt,zRn,zTitle);
+  }
+  db_finalize(&q);
+}
+
+/*
+** user defined separator used by ticket show command
+*/
+static const char *zSep = 0;
+
+/*
+** select the quoting algorithm for "ticket show"
+*/
+#if INTERFACE
+typedef enum eTktShowEnc { tktNoTab=0, tktFossilize=1 } tTktShowEncoding;
+#endif
+static tTktShowEncoding tktEncode = tktNoTab;
+
+/*
+** Output the text given in the argument.  Convert tabs and newlines into
+** spaces.
+*/
+static void output_no_tabs_file(const char *z){
+  switch( tktEncode ){
+    case tktFossilize:
+      { char *zFosZ;
+
+        if( z && *z ){
+          zFosZ = fossilize(z,-1);
+          printf("%s",zFosZ);
+          free(zFosZ);
+        }
+        break;
+      }
+    default:
+      while( z && z[0] ){
+        int i, j;
+        for(i=0; z[i] && (!fossil_isspace(z[i]) || z[i]==' '); i++){}
+        if( i>0 ){
+          printf("%.*s", i, z);
+        }
+        for(j=i; fossil_isspace(z[j]); j++){}
+        if( j>i ){
+          printf("%*s", j-i, "");
+        }
+        z += j;
+      }
+      break; 
+  }
+}
+
+/*
+** Output a row as a tab-separated line of text.
+*/
+int output_separated_file(
+  void *pUser,     /* Pointer to row-count integer */
+  int nArg,        /* Number of columns in this result row */
+  char **azArg,    /* Text of data in all columns */
+  char **azName    /* Names of the columns */
+){
+  int *pCount = (int*)pUser;
+  int i;
+
+  if( *pCount==0 ){
+    for(i=0; i
 #include "rss.h"
 #include 
-#include 
 
 /*
 ** WEBPAGE: timeline.rss
 */
 void page_timeline_rss(void){

Index: src/schema.c
==================================================================
--- src/schema.c
+++ src/schema.c
@@ -241,10 +241,20 @@
 @ -- UUID but we do not (yet) know the file content.
 @ --
 @ CREATE TABLE phantom(
 @   rid INTEGER PRIMARY KEY         -- Record ID of the phantom
 @ );
+@
+@ -- A record of orphaned delta-manifests.  An orphan is a delta-manifest
+@ -- for which we have content, but its baseline-manifest is a phantom.
+@ -- We have to track all orphan maniftests so that when the baseline arrives,
+@ -- we know to process the orphaned deltas.
+@ CREATE TABLE orphan(
+@   rid INTEGER PRIMARY KEY,        -- Delta manifest with a phantom baseline
+@   baseline INTEGER                -- Phantom baseline of this orphan
+@ );
+@ CREATE INDEX orphan_baseline ON orphan(baseline);
 @
 @ -- Unclustered records.  An unclustered record is a record (including
 @ -- a cluster records themselves) that is not mentioned by some other
 @ -- cluster.
 @ --

Index: src/search.c
==================================================================
--- src/search.c
+++ src/search.c
@@ -42,20 +42,19 @@
   int nPattern = strlen(zPattern);
   Search *p;
   char *z;
   int i;
 
-  p = malloc( nPattern + sizeof(*p) + 1);
-  if( p==0 ) fossil_panic("out of memory");
+  p = fossil_malloc( nPattern + sizeof(*p) + 1);
   z = (char*)&p[1];
   strcpy(z, zPattern);
   memset(p, 0, sizeof(*p));
   while( *z && p->nTerma)/sizeof(p->a[0]) ){
-    while( !isalnum(*z) && *z ){ z++; }
+    while( !fossil_isalnum(*z) && *z ){ z++; }
     if( *z==0 ) break;
     p->a[p->nTerm].z = z;
-    for(i=1; isalnum(z[i]) || z[i]=='_'; i++){}
+    for(i=1; fossil_isalnum(z[i]) || z[i]=='_'; i++){}
     p->a[p->nTerm].n = i;
     z += i;
     p->nTerm++;
   }
   return p;

Index: src/setup.c
==================================================================
--- src/setup.c
+++ src/setup.c
@@ -19,10 +19,17 @@
 */
 #include 
 #include "config.h"
 #include "setup.h"
 
+/*
+** The table of web pages supported by this application is generated
+** automatically by the "mkindex" program and written into a file
+** named "page_index.h".  We include that file here to get access
+** to the table.
+*/
+#include "page_index.h"
 
 /*
 ** Output a single entry for a menu generated using an HTML table.
 ** If zLink is not NULL or an empty string, then it is the page that
 ** the menu entry will hyperlink to.  If zLink is NULL or "", then
@@ -57,10 +64,12 @@
     "Grant privileges to individual users.");
   setup_menu_entry("Access", "setup_access",
     "Control access settings.");
   setup_menu_entry("Configuration", "setup_config",
     "Configure the WWW components of the repository");
+  setup_menu_entry("Settings", "setup_settings",
+    "Web interface to the \"fossil settings\" command");
   setup_menu_entry("Timeline", "setup_timeline",
     "Timeline display preferences");
   setup_menu_entry("Tickets", "tktsetup",
     "Configure the trouble-ticketing system for this repository");
   setup_menu_entry("Skins", "setup_skin",
@@ -99,42 +108,40 @@
     return;
   }
 
   style_submenu_element("Add", "Add User", "setup_uedit");
   style_header("User List");
-  @ 
-  @ 
-  @ Users:
-  @ 
-  @ 
+  @ 
+  @ 
+  @ Users:
+  @ 
   @ 
-  @   
-  @   
-  @   
+  @   
+  @   
+  @   
   @ 
   db_prepare(&s, "SELECT uid, login, cap, info FROM user ORDER BY login");
   while( db_step(&s)==SQLITE_ROW ){
     const char *zCap = db_column_text(&s, 2);
     if( strstr(zCap, "s") ) zCap = "s";
     @ 
-    @ 
-    @ 
-    @ 
-    @ 
+    @ 
+    @ 
+    @ 
     @ 
   }
-  @ User ID   Capabilities   Contact Info User ID Capabilities Contact Info

+    @ 
     if( g.okAdmin && (zCap[0]!='s' || g.okSetup) ){
       @ 
     }
-    @ %h(db_column_text(&s,1))
+    @ %h(db_column_text(&s,1))
     if( g.okAdmin ){
       @ 
     }
-    @     %s(zCap)     %s(db_column_text(&s,3)) %s(zCap) %s(db_column_text(&s,3))
-  @ 
-  @ Notes:
+  @ 
+  @ 
+  @ Notes:
   @ 
   @ The permission flags are as follows:
   @ 
      @ 
      @   
@@ -181,31 +188,35 @@
      @   user developer
      @ 
      @   
      @ 
      @   
+     @   /zip URL even without 
+     @    checkout
+     @    and history permissions
   @ a Admin: Create and delete users
w Write-Tkt: Edit tickets
z Zip download: Download a baseline via the
-     @   /zip URL even without checkout
-     @    and history permissions
   @ 
   @
   @ 
-  @ Every user, logged in or not, inherits the privileges of nobody.
+  @ Every user, logged in or not, inherits the privileges of
+  @ nobody.
   @ 
   @
   @ 
-  @ Any human can login as anonymous since the password is
-  @ clearly displayed on the login page for them to type.  The purpose
-  @ of requiring anonymous to log in is to prevent access by spiders.
+  @ Any human can login as anonymous since the
+  @ password is clearly displayed on the login page for them to type. The
+  @ purpose of requiring anonymous to log in is to prevent access by spiders.
   @ Every logged-in user inherits the combined privileges of
-  @ anonymous and
-  @ nobody.
+  @ anonymous and
+  @ nobody.
   @ 
   @
   @ 
-  @ Users with privilege v inherit the combined privileges of
-  @ developer, anonymous, and nobody.
+  @ Users with privilege v inherit the combined
+  @ privileges of developer,
+  @ anonymous, and
+  @ nobody.
   @ 
   @
   @ 
   @ 
   style_footer();
@@ -323,12 +334,12 @@
     }
     if( uid>0 &&
         db_exists("SELECT 1 FROM user WHERE login=%Q AND uid!=%d", zLogin, uid)
     ){
       style_header("User Creation Error");
-      @ Login "%h(zLogin)" is already used by a different
-      @ user.
+      @ Login "%h(zLogin)" is already used by
+      @ a different user.
       @
       @ [Bummer]
       style_footer();
       return;
     }
@@ -353,65 +364,69 @@
   if( uid ){
     zLogin = db_text("", "SELECT login FROM user WHERE uid=%d", uid);
     zInfo = db_text("", "SELECT info FROM user WHERE uid=%d", uid);
     zCap = db_text("", "SELECT cap FROM user WHERE uid=%d", uid);
     zPw = db_text("", "SELECT pw FROM user WHERE uid=%d", uid);
-    if( strchr(zCap, 'a') ) oaa = " checked";
-    if( strchr(zCap, 'b') ) oab = " checked";
-    if( strchr(zCap, 'c') ) oac = " checked";
-    if( strchr(zCap, 'd') ) oad = " checked";
-    if( strchr(zCap, 'e') ) oae = " checked";
-    if( strchr(zCap, 'f') ) oaf = " checked";
-    if( strchr(zCap, 'g') ) oag = " checked";
-    if( strchr(zCap, 'h') ) oah = " checked";
-    if( strchr(zCap, 'i') ) oai = " checked";
-    if( strchr(zCap, 'j') ) oaj = " checked";
-    if( strchr(zCap, 'k') ) oak = " checked";
-    if( strchr(zCap, 'm') ) oam = " checked";
-    if( strchr(zCap, 'n') ) oan = " checked";
-    if( strchr(zCap, 'o') ) oao = " checked";
-    if( strchr(zCap, 'p') ) oap = " checked";
-    if( strchr(zCap, 'r') ) oar = " checked";
-    if( strchr(zCap, 's') ) oas = " checked";
-    if( strchr(zCap, 't') ) oat = " checked";
-    if( strchr(zCap, 'u') ) oau = " checked";
-    if( strchr(zCap, 'v') ) oav = " checked";
-    if( strchr(zCap, 'w') ) oaw = " checked";
-    if( strchr(zCap, 'z') ) oaz = " checked";
+    if( strchr(zCap, 'a') ) oaa = " checked=\"checked\"";
+    if( strchr(zCap, 'b') ) oab = " checked=\"checked\"";
+    if( strchr(zCap, 'c') ) oac = " checked=\"checked\"";
+    if( strchr(zCap, 'd') ) oad = " checked=\"checked\"";
+    if( strchr(zCap, 'e') ) oae = " checked=\"checked\"";
+    if( strchr(zCap, 'f') ) oaf = " checked=\"checked\"";
+    if( strchr(zCap, 'g') ) oag = " checked=\"checked\"";
+    if( strchr(zCap, 'h') ) oah = " checked=\"checked\"";
+    if( strchr(zCap, 'i') ) oai = " checked=\"checked\"";
+    if( strchr(zCap, 'j') ) oaj = " checked=\"checked\"";
+    if( strchr(zCap, 'k') ) oak = " checked=\"checked\"";
+    if( strchr(zCap, 'm') ) oam = " checked=\"checked\"";
+    if( strchr(zCap, 'n') ) oan = " checked=\"checked\"";
+    if( strchr(zCap, 'o') ) oao = " checked=\"checked\"";
+    if( strchr(zCap, 'p') ) oap = " checked=\"checked\"";
+    if( strchr(zCap, 'r') ) oar = " checked=\"checked\"";
+    if( strchr(zCap, 's') ) oas = " checked=\"checked\"";
+    if( strchr(zCap, 't') ) oat = " checked=\"checked\"";
+    if( strchr(zCap, 'u') ) oau = " checked=\"checked\"";
+    if( strchr(zCap, 'v') ) oav = " checked=\"checked\"";
+    if( strchr(zCap, 'w') ) oaw = " checked=\"checked\"";
+    if( strchr(zCap, 'z') ) oaz = " checked=\"checked\"";
   }
 
   /* figure out inherited permissions */
   memset(inherit, 0, sizeof(inherit));
   if( strcmp(zLogin, "developer") ){
     char *z1, *z2;
     z1 = z2 = db_text(0,"SELECT cap FROM user WHERE login='developer'");
     while( z1 && *z1 ){
-      inherit[0x7f & *(z1++)] = "•";
+      inherit[0x7f & *(z1++)] =
+         "•";
     }
     free(z2);
   }
   if( strcmp(zLogin, "reader") ){
     char *z1, *z2;
     z1 = z2 = db_text(0,"SELECT cap FROM user WHERE login='reader'");
     while( z1 && *z1 ){
-      inherit[0x7f & *(z1++)] = "•";
+      inherit[0x7f & *(z1++)] =
+          "•";
     }
     free(z2);
   }
   if( strcmp(zLogin, "anonymous") ){
     char *z1, *z2;
     z1 = z2 = db_text(0,"SELECT cap FROM user WHERE login='anonymous'");
     while( z1 && *z1 ){
-      inherit[0x7f & *(z1++)] = "•";
+      inherit[0x7f & *(z1++)] =
+           "•";
     }
     free(z2);
   }
   if( strcmp(zLogin, "nobody") ){
     char *z1, *z2;
     z1 = z2 = db_text(0,"SELECT cap FROM user WHERE login='nobody'");
     while( z1 && *z1 ){
-      inherit[0x7f & *(z1++)] = "•";
+      inherit[0x7f & *(z1++)] =
+           "•";
     }
     free(z2);
   }
 
   /* Begin generating the page
@@ -420,172 +435,193 @@
   if( uid ){
     style_header(mprintf("Edit User %h", zLogin));
   }else{
     style_header("Add A New User");
   }
-  @ 
-  @ 
+  @ 
+  @ 
   login_insert_csrf_secret();
   @ 
   @ 
-  @   
-  if( uid ){
-    @   
-  }else{
-    @   
-  }
-  @ 
-  @ 
-  @   
-  @   
-  @ 
-  @ 
-  @   
-  @   
-  @ 
-  @ 
-  @   
+  @   
+  if( uid ){
+    @   
+  }else{
+    @   
+  }
+  @ 
+  @ 
+  @   
+  @   
+  @ 
+  @ 
+  @   
+  @   
+  @ 
+  @ 
+  @   
   @   
   @ 
   @ 
   @   
   if( zPw[0] ){
     /* Obscure the password for all users */
-    @   
+    @   
   }else{
     /* Show an empty password as an empty input field */
-    @   
+    @   
   }
   @ 
   if( !higherUser ){
     @ 
-    @   
-    @   
+    @   
     @ 
   }
-  @ User ID: %d(uid) (new user)
Login: 
Contact Info: 
Capabilities: User ID: %d(uid) (new user)
Login: 
Contact Info: 
Capabilities: 
 #define B(x) inherit[x]
   if( g.okSetup ){
-    @    %s(B('s'))Setup

-  }
-  @    %s(B('a'))Admin

-  @    %s(B('d'))Delete

-  @    %s(B('e'))Email

-  @    %s(B('p'))Password

-  @    %s(B('i'))Check-In

-  @    %s(B('o'))Check-Out

-  @    %s(B('h'))History

-  @    %s(B('u'))Reader

-  @    %s(B('v'))Developer

-  @    %s(B('g'))Clone

-  @    %s(B('j'))Read Wiki

-  @    %s(B('f'))New Wiki

-  @    %s(B('m'))Append Wiki

-  @    %s(B('k'))Write Wiki

-  @    %s(B('b'))Attachments

-  @    %s(B('r'))Read Ticket

-  @    %s(B('n'))New Ticket

-  @    %s(B('c'))Append Ticket

-  @    %s(B('w'))Write Ticket

-  @    %s(B('t'))Ticket Report

-  @    %s(B('z'))Download Zip
+    @    %s(B('s'))Setup

+  }
+  @    %s(B('a'))Admin

+  @    %s(B('d'))Delete

+  @    %s(B('e'))Email

+  @    %s(B('p'))Password

+  @    %s(B('i'))Check-In

+  @    %s(B('o'))Check-Out

+  @    %s(B('h'))History

+  @    %s(B('u'))Reader

+  @    %s(B('v'))Developer

+  @    %s(B('g'))Clone

+  @    %s(B('j'))Read Wiki

+  @    %s(B('f'))New Wiki

+  @    %s(B('m'))Append Wiki

+  @    %s(B('k'))Write Wiki

+  @    %s(B('b'))Attachments

+  @    %s(B('r'))Read Ticket

+  @    %s(B('n'))New Ticket

+  @    %s(B('c'))Append Ticket

+  @    %s(B('w'))Write Ticket

+  @    %s(B('t'))Ticket Report

+  @    %s(B('z'))Download Zip
   @   
Password: 
  
+    @     
+  @ 
+  @ 
+  @ 
   @ Privileges And Capabilities:
   @ 
   if( higherUser ){
-    @ 
+    @ 

     @ User %h(zLogin) has Setup privileges and you only have Admin privileges
     @ so you are not permitted to make changes to %h(zLogin).
-    @ 
+    @ 
     @
   }
   @ 
-  @ The Setup user can make arbitrary configuration changes.
-  @ An Admin user can add other users and change user privileges
+  @ The Setup user can make arbitrary
+  @ configuration changes. An Admin user
+  @ can add other users and change user privileges
   @ and reset user passwords.  Both automatically get all other privileges
   @ listed below.  Use these two settings with discretion.
   @ 
   @
   @ 
-  @ The "•" mark indicates
-  @ the privileges of "nobody" that are available to all users
-  @ regardless of whether or not they are logged in.
-  @ 
-  @
-  @ 
-  @ The "•" mark indicates
-  @ the privileges of "anonymous" that are inherited by all logged-in users.
-  @ 
-  @
-  @ 
-  @ The "•" mark indicates
-  @ the privileges of "developer" that are inherited by all users with
-  @ the Developer privilege.
-  @ 
-  @
-  @ 
-  @ The "•" mark indicates
-  @ the privileges of "reader" that are inherited by all users with
-  @ the Reader privilege.
-  @ 
-  @
-  @ 
-  @ The Delete privilege give the user the ability to erase
-  @ wiki, tickets, and attachments that have been added by anonymous
-  @ users.  This capability is intended for deletion of spam.  The
-  @ delete capability is only in effect for 24 hours after the item
-  @ is first posted.  The Setup user can delete anything at any time.
-  @ 
-  @
-  @ 
-  @ The History privilege allows a user to see most hyperlinks.
-  @ This is recommended ON for most logged-in users but OFF for
-  @ user "nobody" to avoid problems with spiders trying to walk every
-  @ historical version of every baseline and file.
-  @ 
-  @
-  @ 
-  @ The Zip privilege allows a user to see the "download as ZIP"
-  @ hyperlink and permits access to the /zip page.  This allows
-  @ users to download ZIP archives without granting other rights like
-  @ Read or History.  This privilege is recommended for
-  @ user nobody so that automatic package downloaders can obtain
-  @ the sources without going through the login procedure.
-  @ 
-  @
-  @ 
-  @ The Check-in privilege allows remote users to "push".
-  @ The Check-out privilege allows remote users to "pull".
-  @ The Clone privilege allows remote users to "clone".
-  @ 

-  @
-  @ 

-  @ The Read Wiki, New Wiki, Append Wiki, and
-  @ Write Wiki privileges control access to wiki pages.  The
-  @ Read Ticket, New Ticket, Append Ticket, and
-  @ Write Ticket privileges control access to trouble tickets.
-  @ The Ticket Report privilege allows the user to create or edit
-  @ ticket report formats.
-  @ 
-  @
-  @ 
-  @ Users with the Password privilege are allowed to change their
-  @ own password.  Recommended ON for most users but OFF for special
-  @ users "developer", "anonymous", and "nobody".
-  @ 
-  @
-  @ 
-  @ The EMail privilege allows the display of sensitive information
-  @ such as the email address of users and contact information on tickets.
-  @ Recommended OFF for "anonymous" and for "nobody" but ON for
-  @ "developer".
-  @ 
-  @
-  @ 
-  @ The Attachment privilege is needed in order to add attachments
-  @ to tickets or wiki.  Write privilege on the ticket or wiki is also
-  @ required.
+  @ The "•" mark
+  @ indicates the privileges of nobody that
+  @ are available to all users regardless of whether or not they are logged in.
+  @ 
+  @
+  @ 
+  @ The "•" mark
+  @ indicates the privileges of anonymous that
+  @ are inherited by all logged-in users.
+  @ 
+  @
+  @ 
+  @ The "•" mark
+  @ indicates the privileges of developer that
+  @ are inherited by all users with the
+  @ Developer privilege.
+  @ 
+  @
+  @ 
+  @ The "•" mark
+  @ indicates the privileges of reader that
+  @ are inherited by all users with the Reader
+  @ privilege.
+  @ 
+  @
+  @ 
+  @ The Delete privilege give the user the
+  @ ability to erase wiki, tickets, and attachments that have been added
+  @ by anonymous users.  This capability is intended for deletion of spam. 
+  @ The delete capability is only in effect for 24 hours after the item
+  @ is first posted.  The Setup user can
+  @ delete anything at any time.
+  @ 
+  @
+  @ 
+  @ The History privilege allows a user
+  @ to see most hyperlinks. This is recommended ON for most logged-in users
+  @ but OFF for user "nobody" to avoid problems with spiders trying to walk
+  @ every historical version of every baseline and file.
+  @ 
+  @
+  @ 
+  @ The Zip privilege allows a user to
+  @ see the "download as ZIP"
+  @ hyperlink and permits access to the /zip page.  This allows
+  @ users to download ZIP archives without granting other rights like
+  @ Read or
+  @ History.  This privilege is recommended for
+  @ user nobody so that automatic package
+  @ downloaders can obtain the sources without going through the login
+  @ procedure.
+  @ 
+  @
+  @ 
+  @ The Check-in privilege allows remote
+  @ users to "push". The Check-out privilege
+  @ allows remote users to "pull". The Clone
+  @ privilege allows remote users to "clone".
+  @ 
+  @
+  @ 
+  @ The Read Wiki,
+  @ New Wiki,
+  @ Append Wiki, and
+  @ Write Wiki privileges control access to wiki pages.  The
+  @ Read Ticket,
+  @ New Ticket,
+  @ Append Ticket, and
+  @ Write Ticket privileges control access
+  @ to trouble tickets.
+  @ The Ticket Report privilege allows
+  @ the user to create or edit ticket report formats.
+  @ 
+  @
+  @ 
+  @ Users with the Password privilege
+  @ are allowed to change their own password.  Recommended ON for most
+  @ users but OFF for special users developer,
+  @ anonymous,
+  @ and nobody.
+  @ 
+  @
+  @ 
+  @ The EMail privilege allows the display of
+  @ sensitive information such as the email address of users and contact
+  @ information on tickets. Recommended OFF for 
+  @ anonymousy and for
+  @ nobody but ON for
+  @ developer.
+  @ 
+  @
+  @ 
+  @ The Attachment privilege is needed in
+  @ order to add attachments to tickets or wiki.  Write privilege on the
+  @ ticket or wiki is also required.
+  @ 
   @
   @ 
   @ Login is prohibited if the password is an empty string.
   @ 
   @ 
@@ -592,42 +628,46 @@
   @
   @ Special Logins
   @
   @ 
   @ 
-  @ No login is required for user "nobody".  The capabilities
-  @ of the nobody user are inherited by all users, regardless of
-  @ whether or not they are logged in.  To disable universal access
-  @ to the repository, make sure no user named "nobody" exists or
-  @ that the nobody user has no capabilities enabled.
-  @ The password for nobody is ignore.  To avoid problems with
-  @ spiders overloading the server, it is recommended
-  @ that the 'h' (History) capability be turned off for the nobody
-  @ user.
-  @ 
-  @
-  @ 
-  @ Login is required for user "anonymous" but the password
-  @ is displayed on the login screen beside the password entry box
-  @ so anybody who can read should be able to login as anonymous.
-  @ On the other hand, spiders and web-crawlers will typically not
-  @ be able to login.  Set the capabilities of the anonymous user
-  @ to things that you want any human to be able to do, but not any
-  @ spider.  Every other logged-in user inherits the privileges of
-  @ anonymous.
-  @ 
-  @
-  @ 
-  @ The "developer" user is intended as a template for trusted users
-  @ with check-in privileges.  When adding new trusted users, simply
-  @ select the Developer privilege to cause the new user to inherit
-  @ all privileges of the "developer" user.  Similarly, the "reader"
-  @ user is a template for users who are allowed more access than anonymous,
-  @ but less than a developer.
-  @ 
-  @ 
-  @ 
+  @ No login is required for user nobody. The
+  @ capabilities of the nobody user are
+  @ inherited by all users, regardless of whether or not they are logged in.
+  @ To disable universal access to the repository, make sure no user named 
+  @ nobody exists or that the
+  @ nobody user has no capabilities
+  @ enabled. The password for nobody is ignore.
+  @ To avoid problems with spiders overloading the server, it is recommended
+  @ that the h (History) capability be turned 
+  @ off for the nobody user.
+  @ 
+  @
+  @ 
+  @ Login is required for user anonymous but the
+  @ password is displayed on the login screen beside the password entry box
+  @ so anybody who can read should be able to login as anonymous.
+  @ On the other hand, spiders and web-crawlers will typically not
+  @ be able to login.  Set the capabilities of the
+  @ anonymous
+  @ user to things that you want any human to be able to do, but not any
+  @ spider.  Every other logged-in user inherits the privileges of
+  @ anonymous.
+  @ 
+  @
+  @ 
+  @ The developer user is intended as a template
+  @ for trusted users with check-in privileges. When adding new trusted users,
+  @ simply select the developer privilege to
+  @ cause the new user to inherit all privileges of the 
+  @ developer
+  @ user.  Similarly, the reader user is a 
+  @ template for users who are allowed more access than
+  @ anonymous,
+  @ but less than a developer.
+  @ 
+  @ 
   style_footer();
 }
 
 
 /*
@@ -651,13 +691,14 @@
       db_set(zVar, iQ ? "1" : "0", 0);
       iVal = iQ;
     }
   }
   if( iVal ){
-    @ %s(zLabel)
+    @ 
+    @ %s(zLabel)
   }else{
-    @ %s(zLabel)
+    @  %s(zLabel)
   }
 }
 
 /*
 ** Generate an entry box for an attribute.
@@ -674,11 +715,11 @@
   if( zQ && strcmp(zQ,zVal)!=0 ){
     login_verify_csrf_secret();
     db_set(zVar, zQ, 0);
     zVal = zQ;
   }
-  @ 
+  @ 
   @ %s(zLabel)
 }
 
 /*
 ** Generate a text box for an attribute.
@@ -698,11 +739,12 @@
     db_set(zVar, zQ, 0);
     z = zQ;
   }
   if( rows>0 && cols>0 ){
     @ %h(z)
-    @ %s(zLabel)
+    if (zLabel && *zLabel)
+      @ %s(zLabel)
   }
 }
 
 
 /*
@@ -714,57 +756,57 @@
     login_needed();
   }
 
   style_header("Access Control Settings");
   db_begin_transaction();
-  @ 
+  @ 
   login_insert_csrf_secret();
-  @ 
+  @ 
   onoff_attribute("Require password for local access",
      "localauth", "localauth", 0);
   @ When enabled, the password sign-in is required for
   @ web access coming from 127.0.0.1.  When disabled, web access
   @ from 127.0.0.1 is allows without any login - the user id is selected
   @ from the ~/.fossil database. Password login is always required
   @ for incoming web connections on internet addresses other than
-  @ 127.0.0.1.
+  @ 127.0.0.1.
 
-  @ 
+  @ 
   onoff_attribute("Allow REMOTE_USER authentication",
      "remote_user_ok", "remote_user_ok", 0);
   @ When enabled, if the REMOTE_USER environment variable is set to the
   @ login name of a valid user and no other login credentials are available,
   @ then the REMOTE_USER is accepted as an authenticated user.
-  @ 
+  @ 
 
-  @ 
+  @ 
   entry_attribute("Login expiration time", 6, "cookie-expire", "cex", "8766");
   @ The number of hours for which a login is valid.  This must be a
   @ positive number.  The default is 8760 hours which is approximately equal
   @ to a year.
 
-  @ 
+  @ 
   entry_attribute("Download packet limit", 10, "max-download", "mxdwn",
                   "5000000");
   @ Fossil tries to limit out-bound sync, clone, and pull packets
   @ to this many bytes, uncompressed.  If the client requires more data
   @ than this, then the client will issue multiple HTTP requests.
   @ Values below 1 million are not recommended.  5 million is a
   @ reasonable number.
 
-  @ 
+  @ 
   onoff_attribute("Show javascript button to fill in CAPTCHA",
                   "auto-captcha", "autocaptcha", 0);
   @ When enabled, a button appears on the login screen for user
   @ "anonymous" that will automatically fill in the CAPTCHA password.
   @ This is less secure that forcing the user to do it manually, but is
   @ probably secure enough and it is certainly more convenient for
   @ anonymous users.
 
-  @ 
-  @ 
-  @ 
+  @ 
+  @ 
+  @ 
   db_end_transaction(0);
   style_footer();
 }
 
 /*
@@ -776,42 +818,88 @@
     login_needed();
   }
 
   style_header("Timeline Display Preferences");
   db_begin_transaction();
-  @ 
+  @ 
   login_insert_csrf_secret();
 
-  @ 
+  @ 
   onoff_attribute("Allow block-markup in timeline",
                   "timeline-block-markup", "tbm", 0);
   @ In timeline displays, check-in comments can be displayed with or
   @ without block markup (paragraphs, tables, etc.)
 
-  @ 
+  @ 
   onoff_attribute("Use Universal Coordinated Time (UTC)",
                   "timeline-utc", "utc", 1);
   @ Show times as UTC (also sometimes called Greenwich Mean Time (GMT) or
   @ Zulu) instead of in local time.
 
-  @ 
+  @ 
   onoff_attribute("Show version differences by default",
                   "show-version-diffs", "vdiff", 0);
   @ On the version-information pages linked from the timeline can either
   @ show complete diffs of all file changes, or can just list the names of
   @ the files that have changed.  Users can get to either page by
   @ clicking.  This setting selects the default.
 
-  @ 
+  @ 
   entry_attribute("Max timeline comment length", 6,
                   "timeline-max-comment", "tmc", "0");
   @ The maximum length of a comment to be displayed in a timeline.
   @ "0" there is no length limit.
 
-  @ 
-  @ 
-  @ 
+  @ 
+  @ 
+  @ 
+  db_end_transaction(0);
+  style_footer();
+}
+
+/*
+** WEBPAGE: setup_settings
+*/
+void setup_settings(void){
+  struct stControlSettings const *pSet;
+
+  login_check_credentials();
+  if( !g.okSetup ){
+    login_needed();
+  }
+
+  style_header("Settings");
+  db_begin_transaction();
+  @ This page provides a simple interface to the "fossil setting" command.
+  @ See the "fossil help setting" output below for further information on
+  @ the meaning of each setting.

+  @ 
+  @ 
+  login_insert_csrf_secret();
+  for(pSet=ctrlSettings; pSet->name!=0; pSet++){
+    if( pSet->width==0 ){
+      onoff_attribute(pSet->name, pSet->name,
+                      pSet->var!=0 ? pSet->var : pSet->name,
+                      is_truth(pSet->def));
+      @ 

+    }
+  }
+  @ 
+  for(pSet=ctrlSettings; pSet->name!=0; pSet++){
+    if( pSet->width!=0 ){
+      entry_attribute(pSet->name, /*pSet->width*/ 40, pSet->name,
+                      pSet->var!=0 ? pSet->var : pSet->name,
+                      (char*)pSet->def);
+      @ 

+    }
+  }
+  @ 
+  @ 
+  @ 
+  @ 

+  @ These settings work in the same way, as the set commandline:

+  @ 
%s(zHelp_setting_cmd)
   db_end_transaction(0);
   style_footer();
 }
 
 /*
@@ -823,11 +911,11 @@
     login_needed();
   }
 
   style_header("WWW Configuration");
   db_begin_transaction();
-  @ 
+  @ 
   login_insert_csrf_secret();
   @ 
   entry_attribute("Project Name", 60, "project-name", "pn", "");
   @ Give your project a name so visitors know what this site is about.
   @ The project name will also be used as the RSS feed title.
@@ -840,36 +928,42 @@
   entry_attribute("Index Page", 60, "index-page", "idxpg", "/home");
   @ Enter the pathname of the page to display when the "Home" menu
   @ option is selected and when no pathname is
   @ specified in the URL.  For example, if you visit the url:
   @
-  @ %h(g.zBaseURL)
+  @ %h(g.zBaseURL)
   @
   @ And you have specified an index page of "/home" the above will
   @ automatically redirect to:
   @
-  @ %h(g.zBaseURL)/home
+  @ %h(g.zBaseURL)/home
   @
   @ The default "/home" page displays a Wiki page with the same name
   @ as the Project Name specified above.  Some sites prefer to redirect
   @ to a documentation page (ex: "/doc/tip/index.wiki") or to "/timeline".
+  @
+  @ Note:  To avoid a redirect loop or other problems, this entry must
+  @ begin with "/" and it must specify a valid page.  For example,
+  @ "/home" will work but "home" will not, since it omits the
+  @ leading "/".
   @ 
   onoff_attribute("Use HTML as wiki markup language",
     "wiki-use-html", "wiki-use-html", 0);
-  @ Use HTML as the wiki markup language. Wiki links will still be parsed but
-  @ all other wiki formatting will be ignored. This option is helpful if you have
-  @ chosen to use a rich HTML editor for wiki markup such as TinyMCE.
+  @ Use HTML as the wiki markup language. Wiki links will still be parsed
+  @ but all other wiki formatting will be ignored. This option is helpful
+  @ if you have chosen to use a rich HTML editor for wiki markup such as
+  @ TinyMCE.
   @ CAUTION: when
   @ enabling, all HTML tags and attributes are accepted in the wiki.
   @ No sanitization is done. This means that it is very possible for malicious
   @ users to inject dangerous HTML, CSS and JavaScript code into your wiki.
   @ This should only be enabled when wiki editing is limited
   @ to trusted users. It should not be used on a publically
   @ editable wiki.
   @ 
-  @ 
-  @ 
+  @ 
+  @ 
   db_end_transaction(0);
   style_footer();
 }
 
 /*
@@ -892,27 +986,27 @@
   if( P("submit")!=0 ){
     db_end_transaction(0);
     cgi_redirect("setup_editcss");
   }
   style_header("Edit CSS");
-  @ 
+  @ 
   login_insert_csrf_secret();
   @ Edit the CSS below:

   textarea_attribute("", 40, 80, "css", "css", zDefaultCSS);
   @ 

-  @ 
-  @ 
-  @ 
-  @ Note: Press your browser Reload button after modifying the
-  @ CSS in order to pull in the modified CSS file.
-  @ 
+  @ 
+  @ 
+  @ 
+  @ Note: Press your browser Reload button after
+  @ modifying the CSS in order to pull in the modified CSS file.
+  @ 
   @ The default CSS is shown below for reference.  Other examples
   @ of CSS files can be seen on the skins page.
   @ See also the header and
   @ footer editing screens.
   @ -  @ %h(zDefaultCSS)
+  cgi_append_default_css();
   @ 
   style_footer();
   db_end_transaction(0);
 }
 
@@ -930,21 +1024,21 @@
     cgi_replace_parameter("header", zDefaultHeader);
   }else{
     textarea_attribute(0, 0, 0, "header", "header", zDefaultHeader);
   }
   style_header("Edit Page Header");
-  @ 
+  @ 
   login_insert_csrf_secret();
   @ Edit HTML text with embedded TH1 (a TCL dialect) that will be used to
   @ generate the beginning of every page through start of the main
   @ menu.
   textarea_attribute("", 40, 80, "header", "header", zDefaultHeader);
   @ 

-  @ 
-  @ 
-  @ 
-  @ 
+  @ 
+  @ 
+  @ 
+  @ 
   @ The default header is shown below for reference.  Other examples
   @ of headers can be seen on the skins page.
   @ See also the CSS and
   @ footer editing screeens.
   @ @@ -968,20 +1062,20 @@
     cgi_replace_parameter("footer", zDefaultFooter);
   }else{
     textarea_attribute(0, 0, 0, "footer", "footer", zDefaultFooter);
   }
   style_header("Edit Page Footer");
-  @ 
+  @ 
   login_insert_csrf_secret();
   @ Edit HTML text with embedded TH1 (a TCL dialect) that will be used to
   @ generate the end of every page.
   textarea_attribute("", 20, 80, "footer", "footer", zDefaultFooter);
   @ 

-  @ 
-  @ 
-  @ 
-  @ 
+  @ 
+  @ 
+  @ 
+  @ 
   @ The default footer is shown below for reference.  Other examples
   @ of footers can be seen on the skins page.
   @ See also the CSS and
   @ header editing screens.
   @ @@ -1031,31 +1125,31 @@
     cgi_redirect("setup_logo");
   }
   style_header("Edit Project Logo");
   @ The current project logo has a MIME-Type of %h(zMime) and looks
   @ like this:
-  @ 
+  @ 
   @
   @ The logo is accessible to all users at this URL:
   @ %s(g.zBaseURL)/logo.
   @ The logo may or may not appear on each
   @ page depending on the CSS and
   @ header setup.
   @
-  @ 
+  @ 
   @ To set a new logo image, select a file to use as the logo using
   @ the entry box below and then press the "Change Logo" button.
   login_insert_csrf_secret();
   @ Logo Image file:
-  @ 

-  @ 
-  @ 
-  @ 
-  @
-  @ Note:  Your browser has probably cached the logo image, so
-  @ you will probably need to press the Reload button on your browser after
-  @ changing the logo to provoke your browser to reload the new logo image.
-  @ 
+  @ 

+  @ 
+  @ 
+  @ 
+  @
+  @ Note:  Your browser has probably cached the
+  @ logo image, so you will probably need to press the Reload button on your
+  @ browser after changing the logo to provoke your browser to reload the new
+  @ logo image. 
   style_footer();
   db_end_transaction(0);
 }

Index: src/sha1.c
==================================================================
--- src/sha1.c
+++ src/sha1.c
@@ -1,415 +1,190 @@
 /*
-** This implementation of SHA1 is adapted from the example implementation
-** contained in RFC-3174.
+** This implementation of SHA1.
 */
-#include 
 #include 
 #include "config.h"
 #include "sha1.h"
 
-/*
- * If you do not have the ISO standard stdint.h header file, then you
- * must typdef the following:
- *    name              meaning
- *  uint32_t         unsigned 32 bit integer
- *  uint8_t          unsigned 8 bit integer (i.e., unsigned char)
- *
- */
-#define SHA1HashSize 20
-#define shaSuccess 0
-#define shaInputTooLong 1
-#define shaStateError 2
-
-/*
- *  This structure will hold context information for the SHA-1
- *  hashing operation
- */
-typedef struct SHA1Context SHA1Context;
-struct SHA1Context {
-    uint32_t Intermediate_Hash[SHA1HashSize/4]; /* Message Digest  */
-
-    uint32_t Length_Low;            /* Message length in bits      */
-    uint32_t Length_High;           /* Message length in bits      */
-
-    int Message_Block_Index;   /* Index into message block array   */
-    uint8_t Message_Block[64];      /* 512-bit message blocks      */
-
-    int Computed;               /* Is the digest computed?         */
-    int Corrupted;             /* Is the message digest corrupted? */
-};
-
-/*
- *  sha1.c
- *
- *  Description:
- *      This file implements the Secure Hashing Algorithm 1 as
- *      defined in FIPS PUB 180-1 published April 17, 1995.
- *
- *      The SHA-1, produces a 160-bit message digest for a given
- *      data stream.  It should take about 2**n steps to find a
- *      message with the same digest as a given message and
- *      2**(n/2) to find any two messages with the same digest,
- *      when n is the digest size in bits.  Therefore, this
- *      algorithm can serve as a means of providing a
- *      "fingerprint" for a message.
- *
- *  Portability Issues:
- *      SHA-1 is defined in terms of 32-bit "words".  This code
- *      uses  (included via "sha1.h" to define 32 and 8
- *      bit unsigned integer types.  If your C compiler does not
- *      support 32 bit unsigned integers, this code is not
- *      appropriate.
- *
- *  Caveats:
- *      SHA-1 is designed to work with messages less than 2^64 bits
- *      long.  Although SHA-1 allows a message digest to be generated
- *      for messages of any number of bits less than 2^64, this
- *      implementation only works with messages with a length that is
- *      a multiple of the size of an 8-bit character.
- *
- */
-
-/*
- *  Define the SHA1 circular left shift macro
- */
-#define SHA1CircularShift(bits,word) \
-                (((word) << (bits)) | ((word) >> (32-(bits))))
-
-/* Local Function Prototyptes */
-static void SHA1PadMessage(SHA1Context *);
-static void SHA1ProcessMessageBlock(SHA1Context *);
-
-/*
- *  SHA1Reset
- *
- *  Description:
- *      This function will initialize the SHA1Context in preparation
- *      for computing a new SHA1 message digest.
- *
- *  Parameters:
- *      context: [in/out]
- *          The context to reset.
- *
- *  Returns:
- *      sha Error Code.
- *
- */
-static int SHA1Reset(SHA1Context *context)
-{
-    context->Length_Low             = 0;
-    context->Length_High            = 0;
-    context->Message_Block_Index    = 0;
-
-    context->Intermediate_Hash[0]   = 0x67452301;
-    context->Intermediate_Hash[1]   = 0xEFCDAB89;
-    context->Intermediate_Hash[2]   = 0x98BADCFE;
-    context->Intermediate_Hash[3]   = 0x10325476;
-    context->Intermediate_Hash[4]   = 0xC3D2E1F0;
-
-    context->Computed   = 0;
-    context->Corrupted  = 0;
-
-    return shaSuccess;
-}
-
-/*
- *  SHA1Result
- *
- *  Description:
- *      This function will return the 160-bit message digest into the
- *      Message_Digest array  provided by the caller.
- *      NOTE: The first octet of hash is stored in the 0th element,
- *            the last octet of hash in the 19th element.
- *
- *  Parameters:
- *      context: [in/out]
- *          The context to use to calculate the SHA-1 hash.
- *      Message_Digest: [out]
- *          Where the digest is returned.
- *
- *  Returns:
- *      sha Error Code.
- *
- */
-static int SHA1Result( SHA1Context *context,
-                uint8_t Message_Digest[SHA1HashSize])
-{
-    int i;
-
-    if (context->Corrupted)
-    {
-        return context->Corrupted;
-    }
-
-    if (!context->Computed)
-    {
-        SHA1PadMessage(context);
-        for(i=0; i<64; ++i)
-        {
-            /* message may be sensitive, clear it out */
-            context->Message_Block[i] = 0;
-        }
-        context->Length_Low = 0;    /* and clear length */
-        context->Length_High = 0;
-        context->Computed = 1;
-
-    }
-
-    for(i = 0; i < SHA1HashSize; ++i)
-    {
-        Message_Digest[i] = context->Intermediate_Hash[i>>2]
-                            >> 8 * ( 3 - ( i & 0x03 ) );
-    }
-
-    return shaSuccess;
-}
-
-/*
- *  SHA1Input
- *
- *  Description:
- *      This function accepts an array of octets as the next portion
- *      of the message.
- *
- *  Parameters:
- *      context: [in/out]
- *          The SHA context to update
- *      message_array: [in]
- *          An array of characters representing the next portion of
- *          the message.
- *      length: [in]
- *          The length of the message in message_array
- *
- *  Returns:
- *      sha Error Code.
- *
- */
-static
-int SHA1Input(    SHA1Context    *context,
-                  const uint8_t  *message_array,
-                  unsigned       length)
-{
-    if (!length)
-    {
-        return shaSuccess;
-    }
-
-    if (context->Computed)
-    {
-        context->Corrupted = shaStateError;
-
-        return shaStateError;
-    }
-
-    if (context->Corrupted)
-    {
-         return context->Corrupted;
-    }
-    while(length-- && !context->Corrupted)
-    {
-    context->Message_Block[context->Message_Block_Index++] =
-                    (*message_array & 0xFF);
-
-    context->Length_Low += 8;
-    if (context->Length_Low == 0)
-    {
-        context->Length_High++;
-        if (context->Length_High == 0)
-        {
-            /* Message is too long */
-            context->Corrupted = 1;
-        }
-    }
-
-    if (context->Message_Block_Index == 64)
-    {
-        SHA1ProcessMessageBlock(context);
-    }
-
-    message_array++;
-    }
-
-    return shaSuccess;
-}
-
-/*
- *  SHA1ProcessMessageBlock
- *
- *  Description:
- *      This function will process the next 512 bits of the message
- *      stored in the Message_Block array.
- *
- *  Parameters:
- *      None.
- *
- *  Returns:
- *      Nothing.
- *
- *  Comments:
- *      Many of the variable names in this code, especially the
- *      single character names, were used because those were the
- *      names used in the publication.
- *
- *
- */
-static void SHA1ProcessMessageBlock(SHA1Context *context)
-{
-    const uint32_t K[] =    {       /* Constants defined in SHA-1   */
-                            0x5A827999,
-                            0x6ED9EBA1,
-                            0x8F1BBCDC,
-                            0xCA62C1D6
-                            };
-    int           t;                 /* Loop counter                */
-    uint32_t      temp;              /* Temporary word value        */
-    uint32_t      W[80];             /* Word sequence               */
-    uint32_t      A, B, C, D, E;     /* Word buffers                */
-
-    /*
-     *  Initialize the first 16 words in the array W
-     */
-    for(t = 0; t < 16; t++)
-    {
-        W[t] = context->Message_Block[t * 4] << 24;
-        W[t] |= context->Message_Block[t * 4 + 1] << 16;
-        W[t] |= context->Message_Block[t * 4 + 2] << 8;
-        W[t] |= context->Message_Block[t * 4 + 3];
-    }
-
-    for(t = 16; t < 80; t++)
-    {
-       W[t] = SHA1CircularShift(1,W[t-3] ^ W[t-8] ^ W[t-14] ^ W[t-16]);
-    }
-
-    A = context->Intermediate_Hash[0];
-    B = context->Intermediate_Hash[1];
-    C = context->Intermediate_Hash[2];
-    D = context->Intermediate_Hash[3];
-    E = context->Intermediate_Hash[4];
-
-    for(t = 0; t < 20; t++)
-    {
-        temp =  SHA1CircularShift(5,A) +
-                ((B & C) | ((~B) & D)) + E + W[t] + K[0];
-        E = D;
-        D = C;
-        C = SHA1CircularShift(30,B);
-
-        B = A;
-        A = temp;
-    }
-
-    for(t = 20; t < 40; t++)
-    {
-        temp = SHA1CircularShift(5,A) + (B ^ C ^ D) + E + W[t] + K[1];
-        E = D;
-        D = C;
-        C = SHA1CircularShift(30,B);
-        B = A;
-        A = temp;
-    }
-
-    for(t = 40; t < 60; t++)
-    {
-        temp = SHA1CircularShift(5,A) +
-               ((B & C) | (B & D) | (C & D)) + E + W[t] + K[2];
-        E = D;
-        D = C;
-        C = SHA1CircularShift(30,B);
-        B = A;
-        A = temp;
-    }
-
-    for(t = 60; t < 80; t++)
-    {
-        temp = SHA1CircularShift(5,A) + (B ^ C ^ D) + E + W[t] + K[3];
-        E = D;
-        D = C;
-        C = SHA1CircularShift(30,B);
-        B = A;
-        A = temp;
-    }
-
-    context->Intermediate_Hash[0] += A;
-    context->Intermediate_Hash[1] += B;
-    context->Intermediate_Hash[2] += C;
-    context->Intermediate_Hash[3] += D;
-    context->Intermediate_Hash[4] += E;
-
-    context->Message_Block_Index = 0;
-}
-
-/*
- *  SHA1PadMessage
- *
-
- *  Description:
- *      According to the standard, the message must be padded to an even
- *      512 bits.  The first padding bit must be a '1'.  The last 64
- *      bits represent the length of the original message.  All bits in
- *      between should be 0.  This function will pad the message
- *      according to those rules by filling the Message_Block array
- *      accordingly.  It will also call the ProcessMessageBlock function
- *      provided appropriately.  When it returns, it can be assumed that
- *      the message digest has been computed.
- *
- *  Parameters:
- *      context: [in/out]
- *          The context to pad
- *      ProcessMessageBlock: [in]
- *          The appropriate SHA*ProcessMessageBlock function
- *  Returns:
- *      Nothing.
- *
- */
-static void SHA1PadMessage(SHA1Context *context)
-{
-    /*
-     *  Check to see if the current message block is too small to hold
-     *  the initial padding bits and length.  If so, we will pad the
-     *  block, process it, and then continue padding into a second
-     *  block.
-     */
-    if (context->Message_Block_Index > 55)
-    {
-        context->Message_Block[context->Message_Block_Index++] = 0x80;
-        while(context->Message_Block_Index < 64)
-        {
-            context->Message_Block[context->Message_Block_Index++] = 0;
-        }
-
-        SHA1ProcessMessageBlock(context);
-
-        while(context->Message_Block_Index < 56)
-        {
-            context->Message_Block[context->Message_Block_Index++] = 0;
-        }
-    }
-    else
-    {
-        context->Message_Block[context->Message_Block_Index++] = 0x80;
-        while(context->Message_Block_Index < 56)
-        {
-
-            context->Message_Block[context->Message_Block_Index++] = 0;
-        }
-    }
-
-    /*
-     *  Store the message length as the last 8 octets
-     */
-    context->Message_Block[56] = context->Length_High >> 24;
-    context->Message_Block[57] = context->Length_High >> 16;
-    context->Message_Block[58] = context->Length_High >> 8;
-    context->Message_Block[59] = context->Length_High;
-    context->Message_Block[60] = context->Length_Low >> 24;
-    context->Message_Block[61] = context->Length_Low >> 16;
-    context->Message_Block[62] = context->Length_Low >> 8;
-    context->Message_Block[63] = context->Length_Low;
-
-    SHA1ProcessMessageBlock(context);
+
+/*
+** The SHA1 implementation below is adapted from:
+**
+**  $NetBSD: sha1.c,v 1.6 2009/11/06 20:31:18 joerg Exp $
+**  $OpenBSD: sha1.c,v 1.9 1997/07/23 21:12:32 kstailey Exp $
+**
+** SHA-1 in C
+** By Steve Reid 
+** 100% Public Domain
+*/
+typedef struct SHA1Context SHA1Context;
+struct SHA1Context {
+  unsigned int state[5];
+  unsigned int count[2];
+  unsigned char buffer[64];
+};
+
+/*
+ * blk0() and blk() perform the initial expand.
+ * I got the idea of expanding during the round function from SSLeay
+ *
+ * blk0le() for little-endian and blk0be() for big-endian.
+ */
+#define rol(value, bits) (((value) << (bits)) | ((value) >> (32 - (bits))))
+#define blk0le(i) (block->l[i] = (rol(block->l[i],24)&0xFF00FF00) \
+    |(rol(block->l[i],8)&0x00FF00FF))
+#define blk0be(i) block->l[i]
+#define blk(i) (block->l[i&15] = rol(block->l[(i+13)&15]^block->l[(i+8)&15] \
+    ^block->l[(i+2)&15]^block->l[i&15],1))
+
+/*
+ * (R0+R1), R2, R3, R4 are the different operations (rounds) used in SHA1
+ *
+ * Rl0() for little-endian and Rb0() for big-endian.  Endianness is 
+ * determined at run-time.
+ */
+#define Rl0(v,w,x,y,z,i) \
+    z+=((w&(x^y))^y)+blk0le(i)+0x5A827999+rol(v,5);w=rol(w,30);
+#define Rb0(v,w,x,y,z,i) \
+    z+=((w&(x^y))^y)+blk0be(i)+0x5A827999+rol(v,5);w=rol(w,30);
+#define R1(v,w,x,y,z,i) \
+    z+=((w&(x^y))^y)+blk(i)+0x5A827999+rol(v,5);w=rol(w,30);
+#define R2(v,w,x,y,z,i) \
+    z+=(w^x^y)+blk(i)+0x6ED9EBA1+rol(v,5);w=rol(w,30);
+#define R3(v,w,x,y,z,i) \
+    z+=(((w|x)&y)|(w&x))+blk(i)+0x8F1BBCDC+rol(v,5);w=rol(w,30);
+#define R4(v,w,x,y,z,i) \
+    z+=(w^x^y)+blk(i)+0xCA62C1D6+rol(v,5);w=rol(w,30);
+
+typedef union {
+    unsigned char c[64];
+    unsigned int l[16];
+} CHAR64LONG16;
+
+/*
+ * Hash a single 512-bit block. This is the core of the algorithm.
+ */
+void SHA1Transform(unsigned int state[5], const unsigned char buffer[64])
+{
+  unsigned int a, b, c, d, e;
+  CHAR64LONG16 *block;
+  static int one = 1;
+  CHAR64LONG16 workspace;
+
+  block = &workspace;
+  (void)memcpy(block, buffer, 64);
+
+  /* Copy context->state[] to working vars */
+  a = state[0];
+  b = state[1];
+  c = state[2];
+  d = state[3];
+  e = state[4];
+
+  /* 4 rounds of 20 operations each. Loop unrolled. */
+  if( 1 == *(unsigned char*)&one ){
+    Rl0(a,b,c,d,e, 0); Rl0(e,a,b,c,d, 1); Rl0(d,e,a,b,c, 2); Rl0(c,d,e,a,b, 3);
+    Rl0(b,c,d,e,a, 4); Rl0(a,b,c,d,e, 5); Rl0(e,a,b,c,d, 6); Rl0(d,e,a,b,c, 7);
+    Rl0(c,d,e,a,b, 8); Rl0(b,c,d,e,a, 9); Rl0(a,b,c,d,e,10); Rl0(e,a,b,c,d,11);
+    Rl0(d,e,a,b,c,12); Rl0(c,d,e,a,b,13); Rl0(b,c,d,e,a,14); Rl0(a,b,c,d,e,15);
+  }else{
+    Rb0(a,b,c,d,e, 0); Rb0(e,a,b,c,d, 1); Rb0(d,e,a,b,c, 2); Rb0(c,d,e,a,b, 3);
+    Rb0(b,c,d,e,a, 4); Rb0(a,b,c,d,e, 5); Rb0(e,a,b,c,d, 6); Rb0(d,e,a,b,c, 7);
+    Rb0(c,d,e,a,b, 8); Rb0(b,c,d,e,a, 9); Rb0(a,b,c,d,e,10); Rb0(e,a,b,c,d,11);
+    Rb0(d,e,a,b,c,12); Rb0(c,d,e,a,b,13); Rb0(b,c,d,e,a,14); Rb0(a,b,c,d,e,15);
+  }
+  R1(e,a,b,c,d,16); R1(d,e,a,b,c,17); R1(c,d,e,a,b,18); R1(b,c,d,e,a,19);
+  R2(a,b,c,d,e,20); R2(e,a,b,c,d,21); R2(d,e,a,b,c,22); R2(c,d,e,a,b,23);
+  R2(b,c,d,e,a,24); R2(a,b,c,d,e,25); R2(e,a,b,c,d,26); R2(d,e,a,b,c,27);
+  R2(c,d,e,a,b,28); R2(b,c,d,e,a,29); R2(a,b,c,d,e,30); R2(e,a,b,c,d,31);
+  R2(d,e,a,b,c,32); R2(c,d,e,a,b,33); R2(b,c,d,e,a,34); R2(a,b,c,d,e,35);
+  R2(e,a,b,c,d,36); R2(d,e,a,b,c,37); R2(c,d,e,a,b,38); R2(b,c,d,e,a,39);
+  R3(a,b,c,d,e,40); R3(e,a,b,c,d,41); R3(d,e,a,b,c,42); R3(c,d,e,a,b,43);
+  R3(b,c,d,e,a,44); R3(a,b,c,d,e,45); R3(e,a,b,c,d,46); R3(d,e,a,b,c,47);
+  R3(c,d,e,a,b,48); R3(b,c,d,e,a,49); R3(a,b,c,d,e,50); R3(e,a,b,c,d,51);
+  R3(d,e,a,b,c,52); R3(c,d,e,a,b,53); R3(b,c,d,e,a,54); R3(a,b,c,d,e,55);
+  R3(e,a,b,c,d,56); R3(d,e,a,b,c,57); R3(c,d,e,a,b,58); R3(b,c,d,e,a,59);
+  R4(a,b,c,d,e,60); R4(e,a,b,c,d,61); R4(d,e,a,b,c,62); R4(c,d,e,a,b,63);
+  R4(b,c,d,e,a,64); R4(a,b,c,d,e,65); R4(e,a,b,c,d,66); R4(d,e,a,b,c,67);
+  R4(c,d,e,a,b,68); R4(b,c,d,e,a,69); R4(a,b,c,d,e,70); R4(e,a,b,c,d,71);
+  R4(d,e,a,b,c,72); R4(c,d,e,a,b,73); R4(b,c,d,e,a,74); R4(a,b,c,d,e,75);
+  R4(e,a,b,c,d,76); R4(d,e,a,b,c,77); R4(c,d,e,a,b,78); R4(b,c,d,e,a,79);
+
+  /* Add the working vars back into context.state[] */
+  state[0] += a;
+  state[1] += b;
+  state[2] += c;
+  state[3] += d;
+  state[4] += e;
+
+  /* Wipe variables */
+  a = b = c = d = e = 0;
+}
+
+
+/*
+ * SHA1Init - Initialize new context
+ */
+static void SHA1Init(SHA1Context *context){
+    /* SHA1 initialization constants */
+    context->state[0] = 0x67452301;
+    context->state[1] = 0xEFCDAB89;
+    context->state[2] = 0x98BADCFE;
+    context->state[3] = 0x10325476;
+    context->state[4] = 0xC3D2E1F0;
+    context->count[0] = context->count[1] = 0;
+}
+
+
+/*
+ * Run your data through this.
+ */
+static void SHA1Update(
+  SHA1Context *context,
+  const unsigned char *data,
+  unsigned int len
+){
+    unsigned int i, j;
+
+    j = context->count[0];
+    if ((context->count[0] += len << 3) < j)
+	context->count[1] += (len>>29)+1;
+    j = (j >> 3) & 63;
+    if ((j + len) > 63) {
+	(void)memcpy(&context->buffer[j], data, (i = 64-j));
+	SHA1Transform(context->state, context->buffer);
+	for ( ; i + 63 < len; i += 64)
+	    SHA1Transform(context->state, &data[i]);
+	j = 0;
+    } else {
+	i = 0;
+    }
+    (void)memcpy(&context->buffer[j], &data[i], len - i);
+}
+
+
+/*
+ * Add padding and return the message digest.
+ */
+static void SHA1Final(SHA1Context *context, unsigned char digest[20]){
+    unsigned int i;
+    unsigned char finalcount[8];
+
+    for (i = 0; i < 8; i++) {
+	finalcount[i] = (unsigned char)((context->count[(i >= 4 ? 0 : 1)]
+	 >> ((3-(i & 3)) * 8) ) & 255);	 /* Endian independent */
+    }
+    SHA1Update(context, (const unsigned char *)"\200", 1);
+    while ((context->count[0] & 504) != 448)
+	SHA1Update(context, (const unsigned char *)"\0", 1);
+    SHA1Update(context, finalcount, 8);  /* Should cause a SHA1Transform() */
+
+    if (digest) {
+	for (i = 0; i < 20; i++)
+	    digest[i] = (unsigned char)
+		((context->state[i>>2] >> ((3-(i & 3)) * 8) ) & 255);
+    }
 }
 
 
 /*
 ** Convert a digest into base-16.  digest should be declared as
@@ -439,18 +214,18 @@
 /*
 ** Add more text to the incremental SHA1 checksum.
 */
 void sha1sum_step_text(const char *zText, int nBytes){
   if( !incrInit ){
-    SHA1Reset(&incrCtx);
+    SHA1Init(&incrCtx);
     incrInit = 1;
   }
   if( nBytes<=0 ){
     if( nBytes==0 ) return;
     nBytes = strlen(zText);
   }
-  SHA1Input(&incrCtx, (unsigned char*)zText, nBytes);
+  SHA1Update(&incrCtx, (unsigned char*)zText, nBytes);
 }
 
 /*
 ** Add the content of a blob to the incremental SHA1 checksum.
 */
@@ -468,11 +243,11 @@
 */
 char *sha1sum_finish(Blob *pOut){
   unsigned char zResult[20];
   static char zOut[41];
   sha1sum_step_text(0,0);
-  SHA1Result(&incrCtx, zResult);
+  SHA1Final(&incrCtx, zResult);
   incrInit = 0;
   DigestToBase16(zResult, zOut);
   if( pOut ){
     blob_zero(pOut);
     blob_append(pOut, zOut, 40);
@@ -495,21 +270,21 @@
 
   in = fopen(zFilename,"rb");
   if( in==0 ){
     return 1;
   }
-  SHA1Reset(&ctx);
+  SHA1Init(&ctx);
   for(;;){
     int n;
     n = fread(zBuf, 1, sizeof(zBuf), in);
     if( n<=0 ) break;
-    SHA1Input(&ctx, (unsigned char*)zBuf, (unsigned)n);
+    SHA1Update(&ctx, (unsigned char*)zBuf, (unsigned)n);
   }
   fclose(in);
   blob_zero(pCksum);
   blob_resize(pCksum, 40);
-  SHA1Result(&ctx, zResult);
+  SHA1Final(&ctx, zResult);
   DigestToBase16(zResult, blob_buffer(pCksum));
   return 0;
 }
 
 /*
@@ -521,19 +296,19 @@
 */
 int sha1sum_blob(const Blob *pIn, Blob *pCksum){
   SHA1Context ctx;
   unsigned char zResult[20];
 
-  SHA1Reset(&ctx);
-  SHA1Input(&ctx, (unsigned char*)blob_buffer(pIn), blob_size(pIn));
+  SHA1Init(&ctx);
+  SHA1Update(&ctx, (unsigned char*)blob_buffer(pIn), blob_size(pIn));
   if( pIn==pCksum ){
     blob_reset(pCksum);
   }else{
     blob_zero(pCksum);
   }
   blob_resize(pCksum, 40);
-  SHA1Result(&ctx, zResult);
+  SHA1Final(&ctx, zResult);
   DigestToBase16(zResult, blob_buffer(pCksum));
   return 0;
 }
 
 /*
@@ -543,13 +318,13 @@
 char *sha1sum(const char *zIn){
   SHA1Context ctx;
   unsigned char zResult[20];
   char zDigest[41];
 
-  SHA1Reset(&ctx);
-  SHA1Input(&ctx, (unsigned const char*)zIn, strlen(zIn));
-  SHA1Result(&ctx, zResult);
+  SHA1Init(&ctx);
+  SHA1Update(&ctx, (unsigned const char*)zIn, strlen(zIn));
+  SHA1Final(&ctx, zResult);
   DigestToBase16(zResult, zDigest);
   return mprintf("%s", zDigest);
 }
 
 /*
@@ -573,11 +348,11 @@
   static char *zProjectId = 0;
   SHA1Context ctx;
   unsigned char zResult[20];
   char zDigest[41];
 
-  SHA1Reset(&ctx);
+  SHA1Init(&ctx);
   if( zProjectId==0 ){
     zProjectId = db_get("project-code", 0);
 
     /* On the first xfer request of a clone, the project-code is not yet
     ** known.  Use the cleartext password, since that is all we have.
@@ -584,16 +359,16 @@
     */
     if( zProjectId==0 ){
       return mprintf("%s", zPw);
     }
   }
-  SHA1Input(&ctx, (unsigned char*)zProjectId, strlen(zProjectId));
-  SHA1Input(&ctx, (unsigned char*)"/", 1);
-  SHA1Input(&ctx, (unsigned char*)zLogin, strlen(zLogin));
-  SHA1Input(&ctx, (unsigned char*)"/", 1);
-  SHA1Input(&ctx, (unsigned const char*)zPw, strlen(zPw));
-  SHA1Result(&ctx, zResult);
+  SHA1Update(&ctx, (unsigned char*)zProjectId, strlen(zProjectId));
+  SHA1Update(&ctx, (unsigned char*)"/", 1);
+  SHA1Update(&ctx, (unsigned char*)zLogin, strlen(zLogin));
+  SHA1Update(&ctx, (unsigned char*)"/", 1);
+  SHA1Update(&ctx, (unsigned const char*)zPw, strlen(zPw));
+  SHA1Final(&ctx, zResult);
   DigestToBase16(zResult, zDigest);
   return mprintf("%s", zDigest);
 }
 
 /*

ADDED   src/shell.c
Index: src/shell.c
==================================================================
--- src/shell.c
+++ src/shell.c
@@ -0,0 +1,2739 @@
+/*
+** 2001 September 15
+**
+** The author disclaims copyright to this source code.  In place of
+** a legal notice, here is a blessing:
+**
+**    May you do good and not evil.
+**    May you find forgiveness for yourself and forgive others.
+**    May you share freely, never taking more than you give.
+**
+*************************************************************************
+** This file contains code to implement the "sqlite" command line
+** utility for accessing SQLite databases.
+*/
+#if defined(_WIN32) || defined(WIN32)
+/* This needs to come before any includes for MSVC compiler */
+#define _CRT_SECURE_NO_WARNINGS
+#endif
+
+#include 
+#include 
+#include 
+#include 
+#include "sqlite3.h"
+#include 
+#include 
+
+#if !defined(_WIN32) && !defined(WIN32) && !defined(__OS2__)
+# include 
+# if !defined(__RTP__) && !defined(_WRS_KERNEL)
+#  include 
+# endif
+# include 
+# include 
+#endif
+
+#ifdef __OS2__
+# include 
+#endif
+
+#if defined(HAVE_READLINE) && HAVE_READLINE==1
+# include 
+# include 
+#else
+# define readline(p) local_getline(p,stdin)
+# define add_history(X)
+# define read_history(X)
+# define write_history(X)
+# define stifle_history(X)
+#endif
+
+#if defined(_WIN32) || defined(WIN32)
+# include 
+#define isatty(h) _isatty(h)
+#define access(f,m) _access((f),(m))
+#else
+/* Make sure isatty() has a prototype.
+*/
+extern int isatty();
+#endif
+
+#if defined(_WIN32_WCE)
+/* Windows CE (arm-wince-mingw32ce-gcc) does not provide isatty()
+ * thus we always assume that we have a console. That can be
+ * overridden with the -batch command line option.
+ */
+#define isatty(x) 1
+#endif
+
+#if !defined(_WIN32) && !defined(WIN32) && !defined(__OS2__) && !defined(__RTP__) && !defined(_WRS_KERNEL)
+#include 
+#include 
+
+/* Saved resource information for the beginning of an operation */
+static struct rusage sBegin;
+
+/* True if the timer is enabled */
+static int enableTimer = 0;
+
+/*
+** Begin timing an operation
+*/
+static void beginTimer(void){
+  if( enableTimer ){
+    getrusage(RUSAGE_SELF, &sBegin);
+  }
+}
+
+/* Return the difference of two time_structs in seconds */
+static double timeDiff(struct timeval *pStart, struct timeval *pEnd){
+  return (pEnd->tv_usec - pStart->tv_usec)*0.000001 + 
+         (double)(pEnd->tv_sec - pStart->tv_sec);
+}
+
+/*
+** Print the timing results.
+*/
+static void endTimer(void){
+  if( enableTimer ){
+    struct rusage sEnd;
+    getrusage(RUSAGE_SELF, &sEnd);
+    printf("CPU Time: user %f sys %f\n",
+       timeDiff(&sBegin.ru_utime, &sEnd.ru_utime),
+       timeDiff(&sBegin.ru_stime, &sEnd.ru_stime));
+  }
+}
+
+#define BEGIN_TIMER beginTimer()
+#define END_TIMER endTimer()
+#define HAS_TIMER 1
+
+#elif (defined(_WIN32) || defined(WIN32))
+
+#include 
+
+/* Saved resource information for the beginning of an operation */
+static HANDLE hProcess;
+static FILETIME ftKernelBegin;
+static FILETIME ftUserBegin;
+typedef BOOL (WINAPI *GETPROCTIMES)(HANDLE, LPFILETIME, LPFILETIME, LPFILETIME, LPFILETIME);
+static GETPROCTIMES getProcessTimesAddr = NULL;
+
+/* True if the timer is enabled */
+static int enableTimer = 0;
+
+/*
+** Check to see if we have timer support.  Return 1 if necessary
+** support found (or found previously).
+*/
+static int hasTimer(void){
+  if( getProcessTimesAddr ){
+    return 1;
+  } else {
+    /* GetProcessTimes() isn't supported in WIN95 and some other Windows versions.
+    ** See if the version we are running on has it, and if it does, save off
+    ** a pointer to it and the current process handle.
+    */
+    hProcess = GetCurrentProcess();
+    if( hProcess ){
+      HINSTANCE hinstLib = LoadLibrary(TEXT("Kernel32.dll"));
+      if( NULL != hinstLib ){
+        getProcessTimesAddr = (GETPROCTIMES) GetProcAddress(hinstLib, "GetProcessTimes");
+        if( NULL != getProcessTimesAddr ){
+          return 1;
+        }
+        FreeLibrary(hinstLib); 
+      }
+    }
+  }
+  return 0;
+}
+
+/*
+** Begin timing an operation
+*/
+static void beginTimer(void){
+  if( enableTimer && getProcessTimesAddr ){
+    FILETIME ftCreation, ftExit;
+    getProcessTimesAddr(hProcess, &ftCreation, &ftExit, &ftKernelBegin, &ftUserBegin);
+  }
+}
+
+/* Return the difference of two FILETIME structs in seconds */
+static double timeDiff(FILETIME *pStart, FILETIME *pEnd){
+  sqlite_int64 i64Start = *((sqlite_int64 *) pStart);
+  sqlite_int64 i64End = *((sqlite_int64 *) pEnd);
+  return (double) ((i64End - i64Start) / 10000000.0);
+}
+
+/*
+** Print the timing results.
+*/
+static void endTimer(void){
+  if( enableTimer && getProcessTimesAddr){
+    FILETIME ftCreation, ftExit, ftKernelEnd, ftUserEnd;
+    getProcessTimesAddr(hProcess, &ftCreation, &ftExit, &ftKernelEnd, &ftUserEnd);
+    printf("CPU Time: user %f sys %f\n",
+       timeDiff(&ftUserBegin, &ftUserEnd),
+       timeDiff(&ftKernelBegin, &ftKernelEnd));
+  }
+}
+
+#define BEGIN_TIMER beginTimer()
+#define END_TIMER endTimer()
+#define HAS_TIMER hasTimer()
+
+#else
+#define BEGIN_TIMER 
+#define END_TIMER
+#define HAS_TIMER 0
+#endif
+
+/*
+** Used to prevent warnings about unused parameters
+*/
+#define UNUSED_PARAMETER(x) (void)(x)
+
+/*
+** If the following flag is set, then command execution stops
+** at an error if we are not interactive.
+*/
+static int bail_on_error = 0;
+
+/*
+** Threat stdin as an interactive input if the following variable
+** is true.  Otherwise, assume stdin is connected to a file or pipe.
+*/
+static int stdin_is_interactive = 1;
+
+/*
+** The following is the open SQLite database.  We make a pointer
+** to this database a static variable so that it can be accessed
+** by the SIGINT handler to interrupt database processing.
+*/
+static sqlite3 *db = 0;
+
+/*
+** True if an interrupt (Control-C) has been received.
+*/
+static volatile int seenInterrupt = 0;
+
+/*
+** This is the name of our program. It is set in main(), used
+** in a number of other places, mostly for error messages.
+*/
+static char *Argv0;
+
+/*
+** Prompt strings. Initialized in main. Settable with
+**   .prompt main continue
+*/
+static char mainPrompt[20];     /* First line prompt. default: "sqlite> "*/
+static char continuePrompt[20]; /* Continuation prompt. default: "   ...> " */
+
+/*
+** Write I/O traces to the following stream.
+*/
+#ifdef SQLITE_ENABLE_IOTRACE
+static FILE *iotrace = 0;
+#endif
+
+/*
+** This routine works like printf in that its first argument is a
+** format string and subsequent arguments are values to be substituted
+** in place of % fields.  The result of formatting this string
+** is written to iotrace.
+*/
+#ifdef SQLITE_ENABLE_IOTRACE
+static void iotracePrintf(const char *zFormat, ...){
+  va_list ap;
+  char *z;
+  if( iotrace==0 ) return;
+  va_start(ap, zFormat);
+  z = sqlite3_vmprintf(zFormat, ap);
+  va_end(ap);
+  fprintf(iotrace, "%s", z);
+  sqlite3_free(z);
+}
+#endif
+
+
+/*
+** Determines if a string is a number of not.
+*/
+static int isNumber(const char *z, int *realnum){
+  if( *z=='-' || *z=='+' ) z++;
+  if( !isdigit(*z) ){
+    return 0;
+  }
+  z++;
+  if( realnum ) *realnum = 0;
+  while( isdigit(*z) ){ z++; }
+  if( *z=='.' ){
+    z++;
+    if( !isdigit(*z) ) return 0;
+    while( isdigit(*z) ){ z++; }
+    if( realnum ) *realnum = 1;
+  }
+  if( *z=='e' || *z=='E' ){
+    z++;
+    if( *z=='+' || *z=='-' ) z++;
+    if( !isdigit(*z) ) return 0;
+    while( isdigit(*z) ){ z++; }
+    if( realnum ) *realnum = 1;
+  }
+  return *z==0;
+}
+
+/*
+** A global char* and an SQL function to access its current value 
+** from within an SQL statement. This program used to use the 
+** sqlite_exec_printf() API to substitue a string into an SQL statement.
+** The correct way to do this with sqlite3 is to use the bind API, but
+** since the shell is built around the callback paradigm it would be a lot
+** of work. Instead just use this hack, which is quite harmless.
+*/
+static const char *zShellStatic = 0;
+static void shellstaticFunc(
+  sqlite3_context *context,
+  int argc,
+  sqlite3_value **argv
+){
+  assert( 0==argc );
+  assert( zShellStatic );
+  UNUSED_PARAMETER(argc);
+  UNUSED_PARAMETER(argv);
+  sqlite3_result_text(context, zShellStatic, -1, SQLITE_STATIC);
+}
+
+
+/*
+** This routine reads a line of text from FILE in, stores
+** the text in memory obtained from malloc() and returns a pointer
+** to the text.  NULL is returned at end of file, or if malloc()
+** fails.
+**
+** The interface is like "readline" but no command-line editing
+** is done.
+*/
+static char *local_getline(char *zPrompt, FILE *in){
+  char *zLine;
+  int nLine;
+  int n;
+  int eol;
+
+  if( zPrompt && *zPrompt ){
+    printf("%s",zPrompt);
+    fflush(stdout);
+  }
+  nLine = 100;
+  zLine = malloc( nLine );
+  if( zLine==0 ) return 0;
+  n = 0;
+  eol = 0;
+  while( !eol ){
+    if( n+100>nLine ){
+      nLine = nLine*2 + 100;
+      zLine = realloc(zLine, nLine);
+      if( zLine==0 ) return 0;
+    }
+    if( fgets(&zLine[n], nLine - n, in)==0 ){
+      if( n==0 ){
+        free(zLine);
+        return 0;
+      }
+      zLine[n] = 0;
+      eol = 1;
+      break;
+    }
+    while( zLine[n] ){ n++; }
+    if( n>0 && zLine[n-1]=='\n' ){
+      n--;
+      if( n>0 && zLine[n-1]=='\r' ) n--;
+      zLine[n] = 0;
+      eol = 1;
+    }
+  }
+  zLine = realloc( zLine, n+1 );
+  return zLine;
+}
+
+/*
+** Retrieve a single line of input text.
+**
+** zPrior is a string of prior text retrieved.  If not the empty
+** string, then issue a continuation prompt.
+*/
+static char *one_input_line(const char *zPrior, FILE *in){
+  char *zPrompt;
+  char *zResult;
+  if( in!=0 ){
+    return local_getline(0, in);
+  }
+  if( zPrior && zPrior[0] ){
+    zPrompt = continuePrompt;
+  }else{
+    zPrompt = mainPrompt;
+  }
+  zResult = readline(zPrompt);
+#if defined(HAVE_READLINE) && HAVE_READLINE==1
+  if( zResult && *zResult ) add_history(zResult);
+#endif
+  return zResult;
+}
+
+struct previous_mode_data {
+  int valid;        /* Is there legit data in here? */
+  int mode;
+  int showHeader;
+  int colWidth[100];
+};
+
+/*
+** An pointer to an instance of this structure is passed from
+** the main program to the callback.  This is used to communicate
+** state and mode information.
+*/
+struct callback_data {
+  sqlite3 *db;           /* The database */
+  int echoOn;            /* True to echo input commands */
+  int statsOn;           /* True to display memory stats before each finalize */
+  int cnt;               /* Number of records displayed so far */
+  FILE *out;             /* Write results here */
+  int mode;              /* An output mode setting */
+  int writableSchema;    /* True if PRAGMA writable_schema=ON */
+  int showHeader;        /* True to show column names in List or Column mode */
+  char *zDestTable;      /* Name of destination table when MODE_Insert */
+  char separator[20];    /* Separator character for MODE_List */
+  int colWidth[100];     /* Requested width of each column when in column mode*/
+  int actualWidth[100];  /* Actual width of each column */
+  char nullvalue[20];    /* The text to print when a NULL comes back from
+                         ** the database */
+  struct previous_mode_data explainPrev;
+                         /* Holds the mode information just before
+                         ** .explain ON */
+  char outfile[FILENAME_MAX]; /* Filename for *out */
+  const char *zDbFilename;    /* name of the database file */
+  sqlite3_stmt *pStmt;   /* Current statement if any. */
+  FILE *pLog;            /* Write log output here */
+};
+
+/*
+** These are the allowed modes.
+*/
+#define MODE_Line     0  /* One column per line.  Blank line between records */
+#define MODE_Column   1  /* One record per line in neat columns */
+#define MODE_List     2  /* One record per line with a separator */
+#define MODE_Semi     3  /* Same as MODE_List but append ";" to each line */
+#define MODE_Html     4  /* Generate an XHTML table */
+#define MODE_Insert   5  /* Generate SQL "insert" statements */
+#define MODE_Tcl      6  /* Generate ANSI-C or TCL quoted elements */
+#define MODE_Csv      7  /* Quote strings, numbers are plain */
+#define MODE_Explain  8  /* Like MODE_Column, but do not truncate data */
+
+static const char *modeDescr[] = {
+  "line",
+  "column",
+  "list",
+  "semi",
+  "html",
+  "insert",
+  "tcl",
+  "csv",
+  "explain",
+};
+
+/*
+** Number of elements in an array
+*/
+#define ArraySize(X)  (int)(sizeof(X)/sizeof(X[0]))
+
+/*
+** Compute a string length that is limited to what can be stored in
+** lower 30 bits of a 32-bit signed integer.
+*/
+static int strlen30(const char *z){
+  const char *z2 = z;
+  while( *z2 ){ z2++; }
+  return 0x3fffffff & (int)(z2 - z);
+}
+
+/*
+** A callback for the sqlite3_log() interface.
+*/
+static void shellLog(void *pArg, int iErrCode, const char *zMsg){
+  struct callback_data *p = (struct callback_data*)pArg;
+  if( p->pLog==0 ) return;
+  fprintf(p->pLog, "(%d) %s\n", iErrCode, zMsg);
+  fflush(p->pLog);
+}
+
+/*
+** Output the given string as a hex-encoded blob (eg. X'1234' )
+*/
+static void output_hex_blob(FILE *out, const void *pBlob, int nBlob){
+  int i;
+  char *zBlob = (char *)pBlob;
+  fprintf(out,"X'");
+  for(i=0; i0 ){
+      fprintf(out,"%.*s",i,z);
+    }
+    if( z[i]=='<' ){
+      fprintf(out,"<");
+    }else if( z[i]=='&' ){
+      fprintf(out,"&");
+    }else if( z[i]=='>' ){
+      fprintf(out,">");
+    }else if( z[i]=='\"' ){
+      fprintf(out,""");
+    }else if( z[i]=='\'' ){
+      fprintf(out,"'");
+    }else{
+      break;
+    }
+    z += i + 1;
+  }
+}
+
+/*
+** If a field contains any character identified by a 1 in the following
+** array, then the string must be quoted for CSV.
+*/
+static const char needCsvQuote[] = {
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 0, 1, 0, 0, 0, 0, 1,   0, 0, 0, 0, 0, 0, 0, 0, 
+  0, 0, 0, 0, 0, 0, 0, 0,   0, 0, 0, 0, 0, 0, 0, 0, 
+  0, 0, 0, 0, 0, 0, 0, 0,   0, 0, 0, 0, 0, 0, 0, 0, 
+  0, 0, 0, 0, 0, 0, 0, 0,   0, 0, 0, 0, 0, 0, 0, 0, 
+  0, 0, 0, 0, 0, 0, 0, 0,   0, 0, 0, 0, 0, 0, 0, 0, 
+  0, 0, 0, 0, 0, 0, 0, 0,   0, 0, 0, 0, 0, 0, 0, 1, 
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+  1, 1, 1, 1, 1, 1, 1, 1,   1, 1, 1, 1, 1, 1, 1, 1,   
+};
+
+/*
+** Output a single term of CSV.  Actually, p->separator is used for
+** the separator, which may or may not be a comma.  p->nullvalue is
+** the null value.  Strings are quoted using ANSI-C rules.  Numbers
+** appear outside of quotes.
+*/
+static void output_csv(struct callback_data *p, const char *z, int bSep){
+  FILE *out = p->out;
+  if( z==0 ){
+    fprintf(out,"%s",p->nullvalue);
+  }else{
+    int i;
+    int nSep = strlen30(p->separator);
+    for(i=0; z[i]; i++){
+      if( needCsvQuote[((unsigned char*)z)[i]] 
+         || (z[i]==p->separator[0] && 
+             (nSep==1 || memcmp(z, p->separator, nSep)==0)) ){
+        i = 0;
+        break;
+      }
+    }
+    if( i==0 ){
+      putc('"', out);
+      for(i=0; z[i]; i++){
+        if( z[i]=='"' ) putc('"', out);
+        putc(z[i], out);
+      }
+      putc('"', out);
+    }else{
+      fprintf(out, "%s", z);
+    }
+  }
+  if( bSep ){
+    fprintf(p->out, "%s", p->separator);
+  }
+}
+
+#ifdef SIGINT
+/*
+** This routine runs when the user presses Ctrl-C
+*/
+static void interrupt_handler(int NotUsed){
+  UNUSED_PARAMETER(NotUsed);
+  seenInterrupt = 1;
+  if( db ) sqlite3_interrupt(db);
+}
+#endif
+
+/*
+** This is the callback routine that the shell
+** invokes for each row of a query result.
+*/
+static int shell_callback(void *pArg, int nArg, char **azArg, char **azCol, int *aiType){
+  int i;
+  struct callback_data *p = (struct callback_data*)pArg;
+
+  switch( p->mode ){
+    case MODE_Line: {
+      int w = 5;
+      if( azArg==0 ) break;
+      for(i=0; iw ) w = len;
+      }
+      if( p->cnt++>0 ) fprintf(p->out,"\n");
+      for(i=0; iout,"%*s = %s\n", w, azCol[i],
+                azArg[i] ? azArg[i] : p->nullvalue);
+      }
+      break;
+    }
+    case MODE_Explain:
+    case MODE_Column: {
+      if( p->cnt++==0 ){
+        for(i=0; icolWidth) ){
+            w = p->colWidth[i];
+          }else{
+            w = 0;
+          }
+          if( w<=0 ){
+            w = strlen30(azCol[i] ? azCol[i] : "");
+            if( w<10 ) w = 10;
+            n = strlen30(azArg && azArg[i] ? azArg[i] : p->nullvalue);
+            if( wactualWidth) ){
+            p->actualWidth[i] = w;
+          }
+          if( p->showHeader ){
+            fprintf(p->out,"%-*.*s%s",w,w,azCol[i], i==nArg-1 ? "\n": "  ");
+          }
+        }
+        if( p->showHeader ){
+          for(i=0; iactualWidth) ){
+               w = p->actualWidth[i];
+            }else{
+               w = 10;
+            }
+            fprintf(p->out,"%-*.*s%s",w,w,"-----------------------------------"
+                   "----------------------------------------------------------",
+                    i==nArg-1 ? "\n": "  ");
+          }
+        }
+      }
+      if( azArg==0 ) break;
+      for(i=0; iactualWidth) ){
+           w = p->actualWidth[i];
+        }else{
+           w = 10;
+        }
+        if( p->mode==MODE_Explain && azArg[i] && 
+           strlen30(azArg[i])>w ){
+          w = strlen30(azArg[i]);
+        }
+        fprintf(p->out,"%-*.*s%s",w,w,
+            azArg[i] ? azArg[i] : p->nullvalue, i==nArg-1 ? "\n": "  ");
+      }
+      break;
+    }
+    case MODE_Semi:
+    case MODE_List: {
+      if( p->cnt++==0 && p->showHeader ){
+        for(i=0; iout,"%s%s",azCol[i], i==nArg-1 ? "\n" : p->separator);
+        }
+      }
+      if( azArg==0 ) break;
+      for(i=0; inullvalue;
+        fprintf(p->out, "%s", z);
+        if( iout, "%s", p->separator);
+        }else if( p->mode==MODE_Semi ){
+          fprintf(p->out, ";\n");
+        }else{
+          fprintf(p->out, "\n");
+        }
+      }
+      break;
+    }
+    case MODE_Html: {
+      if( p->cnt++==0 && p->showHeader ){
+        fprintf(p->out,"

"); + output_html_string(p->out, azCol[i]); + fprintf(p->out,"

"); + output_html_string(p->out, azArg[i] ? azArg[i] : p->nullvalue); + fprintf(p->out,"

code\n" + " insert SQL insert statements for TABLE\n" + " line One value per line\n" + " list Values delimited by .separator string\n" + " tabs Tab-separated values\n" + " tcl TCL list elements\n" + ".nullvalue STRING Print STRING in place of NULL values\n" + ".output FILENAME Send output to FILENAME\n" + ".output stdout Send output to the screen\n" + ".prompt MAIN CONTINUE Replace the standard prompts\n" + ".quit Exit this program\n" + ".read FILENAME Execute SQL in FILENAME\n" + ".restore ?DB? FILE Restore content of DB (default \"main\") from FILE\n" + ".schema ?TABLE? Show the CREATE statements\n" + " If TABLE specified, only show tables matching\n" + " LIKE pattern TABLE.\n" + ".separator STRING Change separator used by output mode and .import\n" + ".show Show the current values for various settings\n" + ".stats ON|OFF Turn stats on or off\n" + ".tables ?TABLE? List names of tables\n" + " If TABLE specified, only list tables matching\n" + " LIKE pattern TABLE.\n" + ".timeout MS Try opening locked tables for MS milliseconds\n" + ".width NUM1 NUM2 ... Set column widths for \"column\" mode\n" +; + +static char zTimerHelp[] = + ".timer ON|OFF Turn the CPU timer measurement on or off\n" +; + +/* Forward reference */ +static int process_input(struct callback_data *p, FILE *in); + +/* +** Make sure the database is open. If it is not, then open it. If +** the database fails to open, print an error message and exit. +*/ +static void open_db(struct callback_data *p){ + if( p->db==0 ){ + sqlite3_open(p->zDbFilename, &p->db); + db = p->db; + if( db && sqlite3_errcode(db)==SQLITE_OK ){ + sqlite3_create_function(db, "shellstatic", 0, SQLITE_UTF8, 0, + shellstaticFunc, 0, 0); + } + if( db==0 || SQLITE_OK!=sqlite3_errcode(db) ){ + fprintf(stderr,"Error: unable to open database \"%s\": %s\n", + p->zDbFilename, sqlite3_errmsg(db)); + exit(1); + } +#ifndef SQLITE_OMIT_LOAD_EXTENSION + sqlite3_enable_load_extension(p->db, 1); +#endif + } +} + +/* +** Do C-language style dequoting. +** +** \t -> tab +** \n -> newline +** \r -> carriage return +** \NNN -> ascii character NNN in octal +** \\ -> backslash +*/ +static void resolve_backslashes(char *z){ + int i, j; + char c; + for(i=j=0; (c = z[i])!=0; i++, j++){ + if( c=='\\' ){ + c = z[++i]; + if( c=='n' ){ + c = '\n'; + }else if( c=='t' ){ + c = '\t'; + }else if( c=='r' ){ + c = '\r'; + }else if( c>='0' && c<='7' ){ + c -= '0'; + if( z[i+1]>='0' && z[i+1]<='7' ){ + i++; + c = (c<<3) + z[i] - '0'; + if( z[i+1]>='0' && z[i+1]<='7' ){ + i++; + c = (c<<3) + z[i] - '0'; + } + } + } + } + z[j] = c; + } + z[j] = 0; +} + +/* +** Interpret zArg as a boolean value. Return either 0 or 1. +*/ +static int booleanValue(char *zArg){ + int val = atoi(zArg); + int j; + for(j=0; zArg[j]; j++){ + zArg[j] = (char)tolower(zArg[j]); + } + if( strcmp(zArg,"on")==0 ){ + val = 1; + }else if( strcmp(zArg,"yes")==0 ){ + val = 1; + } + return val; +} + +/* +** If an input line begins with "." then invoke this routine to +** process that line. +** +** Return 1 on error, 2 to exit, and 0 otherwise. +*/ +static int do_meta_command(char *zLine, struct callback_data *p){ + int i = 1; + int nArg = 0; + int n, c; + int rc = 0; + char *azArg[50]; + + /* Parse the input line into tokens. + */ + while( zLine[i] && nArg=3 && strncmp(azArg[0], "backup", n)==0 && nArg>1 && nArg<4){ + const char *zDestFile; + const char *zDb; + sqlite3 *pDest; + sqlite3_backup *pBackup; + if( nArg==2 ){ + zDestFile = azArg[1]; + zDb = "main"; + }else{ + zDestFile = azArg[2]; + zDb = azArg[1]; + } + rc = sqlite3_open(zDestFile, &pDest); + if( rc!=SQLITE_OK ){ + fprintf(stderr, "Error: cannot open \"%s\"\n", zDestFile); + sqlite3_close(pDest); + return 1; + } + open_db(p); + pBackup = sqlite3_backup_init(pDest, "main", p->db, zDb); + if( pBackup==0 ){ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(pDest)); + sqlite3_close(pDest); + return 1; + } + while( (rc = sqlite3_backup_step(pBackup,100))==SQLITE_OK ){} + sqlite3_backup_finish(pBackup); + if( rc==SQLITE_DONE ){ + rc = 0; + }else{ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(pDest)); + rc = 1; + } + sqlite3_close(pDest); + }else + + if( c=='b' && n>=3 && strncmp(azArg[0], "bail", n)==0 && nArg>1 && nArg<3 ){ + bail_on_error = booleanValue(azArg[1]); + }else + + if( c=='d' && n>1 && strncmp(azArg[0], "databases", n)==0 && nArg==1 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 1; + data.mode = MODE_Column; + data.colWidth[0] = 3; + data.colWidth[1] = 15; + data.colWidth[2] = 58; + data.cnt = 0; + sqlite3_exec(p->db, "PRAGMA database_list; ", callback, &data, &zErrMsg); + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + rc = 1; + } + }else + + if( c=='d' && strncmp(azArg[0], "dump", n)==0 && nArg<3 ){ + char *zErrMsg = 0; + open_db(p); + /* When playing back a "dump", the content might appear in an order + ** which causes immediate foreign key constraints to be violated. + ** So disable foreign-key constraint enforcement to prevent problems. */ + fprintf(p->out, "PRAGMA foreign_keys=OFF;\n"); + fprintf(p->out, "BEGIN TRANSACTION;\n"); + p->writableSchema = 0; + sqlite3_exec(p->db, "PRAGMA writable_schema=ON", 0, 0, 0); + if( nArg==1 ){ + run_schema_dump_query(p, + "SELECT name, type, sql FROM sqlite_master " + "WHERE sql NOT NULL AND type=='table' AND name!='sqlite_sequence'", 0 + ); + run_schema_dump_query(p, + "SELECT name, type, sql FROM sqlite_master " + "WHERE name=='sqlite_sequence'", 0 + ); + run_table_dump_query(p->out, p->db, + "SELECT sql FROM sqlite_master " + "WHERE sql NOT NULL AND type IN ('index','trigger','view')", 0 + ); + }else{ + int i; + for(i=1; iout, p->db, + "SELECT sql FROM sqlite_master " + "WHERE sql NOT NULL" + " AND type IN ('index','trigger','view')" + " AND tbl_name LIKE shellstatic()", 0 + ); + zShellStatic = 0; + } + } + if( p->writableSchema ){ + fprintf(p->out, "PRAGMA writable_schema=OFF;\n"); + p->writableSchema = 0; + } + sqlite3_exec(p->db, "PRAGMA writable_schema=OFF", 0, 0, 0); + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + }else{ + fprintf(p->out, "COMMIT;\n"); + } + }else + + if( c=='e' && strncmp(azArg[0], "echo", n)==0 && nArg>1 && nArg<3 ){ + p->echoOn = booleanValue(azArg[1]); + }else + + if( c=='e' && strncmp(azArg[0], "exit", n)==0 && nArg==1 ){ + rc = 2; + }else + + if( c=='e' && strncmp(azArg[0], "explain", n)==0 && nArg<3 ){ + int val = nArg>=2 ? booleanValue(azArg[1]) : 1; + if(val == 1) { + if(!p->explainPrev.valid) { + p->explainPrev.valid = 1; + p->explainPrev.mode = p->mode; + p->explainPrev.showHeader = p->showHeader; + memcpy(p->explainPrev.colWidth,p->colWidth,sizeof(p->colWidth)); + } + /* We could put this code under the !p->explainValid + ** condition so that it does not execute if we are already in + ** explain mode. However, always executing it allows us an easy + ** was to reset to explain mode in case the user previously + ** did an .explain followed by a .width, .mode or .header + ** command. + */ + p->mode = MODE_Explain; + p->showHeader = 1; + memset(p->colWidth,0,ArraySize(p->colWidth)); + p->colWidth[0] = 4; /* addr */ + p->colWidth[1] = 13; /* opcode */ + p->colWidth[2] = 4; /* P1 */ + p->colWidth[3] = 4; /* P2 */ + p->colWidth[4] = 4; /* P3 */ + p->colWidth[5] = 13; /* P4 */ + p->colWidth[6] = 2; /* P5 */ + p->colWidth[7] = 13; /* Comment */ + }else if (p->explainPrev.valid) { + p->explainPrev.valid = 0; + p->mode = p->explainPrev.mode; + p->showHeader = p->explainPrev.showHeader; + memcpy(p->colWidth,p->explainPrev.colWidth,sizeof(p->colWidth)); + } + }else + + if( c=='h' && (strncmp(azArg[0], "header", n)==0 || + strncmp(azArg[0], "headers", n)==0) && nArg>1 && nArg<3 ){ + p->showHeader = booleanValue(azArg[1]); + }else + + if( c=='h' && strncmp(azArg[0], "help", n)==0 ){ + fprintf(stderr,"%s",zHelp); + if( HAS_TIMER ){ + fprintf(stderr,"%s",zTimerHelp); + } + }else + + if( c=='i' && strncmp(azArg[0], "import", n)==0 && nArg==3 ){ + char *zTable = azArg[2]; /* Insert data into this table */ + char *zFile = azArg[1]; /* The file from which to extract data */ + sqlite3_stmt *pStmt = NULL; /* A statement */ + int nCol; /* Number of columns in the table */ + int nByte; /* Number of bytes in an SQL string */ + int i, j; /* Loop counters */ + int nSep; /* Number of bytes in p->separator[] */ + char *zSql; /* An SQL statement */ + char *zLine; /* A single line of input from the file */ + char **azCol; /* zLine[] broken up into columns */ + char *zCommit; /* How to commit changes */ + FILE *in; /* The input file */ + int lineno = 0; /* Line number of input file */ + + open_db(p); + nSep = strlen30(p->separator); + if( nSep==0 ){ + fprintf(stderr, "Error: non-null separator required for import\n"); + return 1; + } + zSql = sqlite3_mprintf("SELECT * FROM '%q'", zTable); + if( zSql==0 ){ + fprintf(stderr, "Error: out of memory\n"); + return 1; + } + nByte = strlen30(zSql); + rc = sqlite3_prepare(p->db, zSql, -1, &pStmt, 0); + sqlite3_free(zSql); + if( rc ){ + if (pStmt) sqlite3_finalize(pStmt); + fprintf(stderr,"Error: %s\n", sqlite3_errmsg(db)); + return 1; + } + nCol = sqlite3_column_count(pStmt); + sqlite3_finalize(pStmt); + pStmt = 0; + if( nCol==0 ) return 0; /* no columns, no error */ + zSql = malloc( nByte + 20 + nCol*2 ); + if( zSql==0 ){ + fprintf(stderr, "Error: out of memory\n"); + return 1; + } + sqlite3_snprintf(nByte+20, zSql, "INSERT INTO '%q' VALUES(?", zTable); + j = strlen30(zSql); + for(i=1; idb, zSql, -1, &pStmt, 0); + free(zSql); + if( rc ){ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(db)); + if (pStmt) sqlite3_finalize(pStmt); + return 1; + } + in = fopen(zFile, "rb"); + if( in==0 ){ + fprintf(stderr, "Error: cannot open \"%s\"\n", zFile); + sqlite3_finalize(pStmt); + return 1; + } + azCol = malloc( sizeof(azCol[0])*(nCol+1) ); + if( azCol==0 ){ + fprintf(stderr, "Error: out of memory\n"); + fclose(in); + sqlite3_finalize(pStmt); + return 1; + } + sqlite3_exec(p->db, "BEGIN", 0, 0, 0); + zCommit = "COMMIT"; + while( (zLine = local_getline(0, in))!=0 ){ + char *z; + i = 0; + lineno++; + azCol[0] = zLine; + for(i=0, z=zLine; *z && *z!='\n' && *z!='\r'; z++){ + if( *z==p->separator[0] && strncmp(z, p->separator, nSep)==0 ){ + *z = 0; + i++; + if( idb, zCommit, 0, 0, 0); + }else + + if( c=='i' && strncmp(azArg[0], "indices", n)==0 && nArg<3 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 0; + data.mode = MODE_List; + if( nArg==1 ){ + rc = sqlite3_exec(p->db, + "SELECT name FROM sqlite_master " + "WHERE type='index' AND name NOT LIKE 'sqlite_%' " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type='index' " + "ORDER BY 1", + callback, &data, &zErrMsg + ); + }else{ + zShellStatic = azArg[1]; + rc = sqlite3_exec(p->db, + "SELECT name FROM sqlite_master " + "WHERE type='index' AND tbl_name LIKE shellstatic() " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type='index' AND tbl_name LIKE shellstatic() " + "ORDER BY 1", + callback, &data, &zErrMsg + ); + zShellStatic = 0; + } + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + rc = 1; + }else if( rc != SQLITE_OK ){ + fprintf(stderr,"Error: querying sqlite_master and sqlite_temp_master\n"); + rc = 1; + } + }else + +#ifdef SQLITE_ENABLE_IOTRACE + if( c=='i' && strncmp(azArg[0], "iotrace", n)==0 ){ + extern void (*sqlite3IoTrace)(const char*, ...); + if( iotrace && iotrace!=stdout ) fclose(iotrace); + iotrace = 0; + if( nArg<2 ){ + sqlite3IoTrace = 0; + }else if( strcmp(azArg[1], "-")==0 ){ + sqlite3IoTrace = iotracePrintf; + iotrace = stdout; + }else{ + iotrace = fopen(azArg[1], "w"); + if( iotrace==0 ){ + fprintf(stderr, "Error: cannot open \"%s\"\n", azArg[1]); + sqlite3IoTrace = 0; + rc = 1; + }else{ + sqlite3IoTrace = iotracePrintf; + } + } + }else +#endif + +#ifndef SQLITE_OMIT_LOAD_EXTENSION + if( c=='l' && strncmp(azArg[0], "load", n)==0 && nArg>=2 ){ + const char *zFile, *zProc; + char *zErrMsg = 0; + zFile = azArg[1]; + zProc = nArg>=3 ? azArg[2] : 0; + open_db(p); + rc = sqlite3_load_extension(p->db, zFile, zProc, &zErrMsg); + if( rc!=SQLITE_OK ){ + fprintf(stderr, "Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + rc = 1; + } + }else +#endif + + if( c=='l' && strncmp(azArg[0], "log", n)==0 && nArg>=1 ){ + const char *zFile = azArg[1]; + if( p->pLog && p->pLog!=stdout && p->pLog!=stderr ){ + fclose(p->pLog); + p->pLog = 0; + } + if( strcmp(zFile,"stdout")==0 ){ + p->pLog = stdout; + }else if( strcmp(zFile, "stderr")==0 ){ + p->pLog = stderr; + }else if( strcmp(zFile, "off")==0 ){ + p->pLog = 0; + }else{ + p->pLog = fopen(zFile, "w"); + if( p->pLog==0 ){ + fprintf(stderr, "Error: cannot open \"%s\"\n", zFile); + } + } + }else + + if( c=='m' && strncmp(azArg[0], "mode", n)==0 && nArg==2 ){ + int n2 = strlen30(azArg[1]); + if( (n2==4 && strncmp(azArg[1],"line",n2)==0) + || + (n2==5 && strncmp(azArg[1],"lines",n2)==0) ){ + p->mode = MODE_Line; + }else if( (n2==6 && strncmp(azArg[1],"column",n2)==0) + || + (n2==7 && strncmp(azArg[1],"columns",n2)==0) ){ + p->mode = MODE_Column; + }else if( n2==4 && strncmp(azArg[1],"list",n2)==0 ){ + p->mode = MODE_List; + }else if( n2==4 && strncmp(azArg[1],"html",n2)==0 ){ + p->mode = MODE_Html; + }else if( n2==3 && strncmp(azArg[1],"tcl",n2)==0 ){ + p->mode = MODE_Tcl; + }else if( n2==3 && strncmp(azArg[1],"csv",n2)==0 ){ + p->mode = MODE_Csv; + sqlite3_snprintf(sizeof(p->separator), p->separator, ","); + }else if( n2==4 && strncmp(azArg[1],"tabs",n2)==0 ){ + p->mode = MODE_List; + sqlite3_snprintf(sizeof(p->separator), p->separator, "\t"); + }else if( n2==6 && strncmp(azArg[1],"insert",n2)==0 ){ + p->mode = MODE_Insert; + set_table_name(p, "table"); + }else { + fprintf(stderr,"Error: mode should be one of: " + "column csv html insert line list tabs tcl\n"); + rc = 1; + } + }else + + if( c=='m' && strncmp(azArg[0], "mode", n)==0 && nArg==3 ){ + int n2 = strlen30(azArg[1]); + if( n2==6 && strncmp(azArg[1],"insert",n2)==0 ){ + p->mode = MODE_Insert; + set_table_name(p, azArg[2]); + }else { + fprintf(stderr, "Error: invalid arguments: " + " \"%s\". Enter \".help\" for help\n", azArg[2]); + rc = 1; + } + }else + + if( c=='n' && strncmp(azArg[0], "nullvalue", n)==0 && nArg==2 ) { + sqlite3_snprintf(sizeof(p->nullvalue), p->nullvalue, + "%.*s", (int)ArraySize(p->nullvalue)-1, azArg[1]); + }else + + if( c=='o' && strncmp(azArg[0], "output", n)==0 && nArg==2 ){ + if( p->out!=stdout ){ + fclose(p->out); + } + if( strcmp(azArg[1],"stdout")==0 ){ + p->out = stdout; + sqlite3_snprintf(sizeof(p->outfile), p->outfile, "stdout"); + }else{ + p->out = fopen(azArg[1], "wb"); + if( p->out==0 ){ + fprintf(stderr,"Error: cannot write to \"%s\"\n", azArg[1]); + p->out = stdout; + rc = 1; + } else { + sqlite3_snprintf(sizeof(p->outfile), p->outfile, "%s", azArg[1]); + } + } + }else + + if( c=='p' && strncmp(azArg[0], "prompt", n)==0 && (nArg==2 || nArg==3)){ + if( nArg >= 2) { + strncpy(mainPrompt,azArg[1],(int)ArraySize(mainPrompt)-1); + } + if( nArg >= 3) { + strncpy(continuePrompt,azArg[2],(int)ArraySize(continuePrompt)-1); + } + }else + + if( c=='q' && strncmp(azArg[0], "quit", n)==0 && nArg==1 ){ + rc = 2; + }else + + if( c=='r' && n>=3 && strncmp(azArg[0], "read", n)==0 && nArg==2 ){ + FILE *alt = fopen(azArg[1], "rb"); + if( alt==0 ){ + fprintf(stderr,"Error: cannot open \"%s\"\n", azArg[1]); + rc = 1; + }else{ + rc = process_input(p, alt); + fclose(alt); + } + }else + + if( c=='r' && n>=3 && strncmp(azArg[0], "restore", n)==0 && nArg>1 && nArg<4){ + const char *zSrcFile; + const char *zDb; + sqlite3 *pSrc; + sqlite3_backup *pBackup; + int nTimeout = 0; + + if( nArg==2 ){ + zSrcFile = azArg[1]; + zDb = "main"; + }else{ + zSrcFile = azArg[2]; + zDb = azArg[1]; + } + rc = sqlite3_open(zSrcFile, &pSrc); + if( rc!=SQLITE_OK ){ + fprintf(stderr, "Error: cannot open \"%s\"\n", zSrcFile); + sqlite3_close(pSrc); + return 1; + } + open_db(p); + pBackup = sqlite3_backup_init(p->db, zDb, pSrc, "main"); + if( pBackup==0 ){ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(p->db)); + sqlite3_close(pSrc); + return 1; + } + while( (rc = sqlite3_backup_step(pBackup,100))==SQLITE_OK + || rc==SQLITE_BUSY ){ + if( rc==SQLITE_BUSY ){ + if( nTimeout++ >= 3 ) break; + sqlite3_sleep(100); + } + } + sqlite3_backup_finish(pBackup); + if( rc==SQLITE_DONE ){ + rc = 0; + }else if( rc==SQLITE_BUSY || rc==SQLITE_LOCKED ){ + fprintf(stderr, "Error: source database is busy\n"); + rc = 1; + }else{ + fprintf(stderr, "Error: %s\n", sqlite3_errmsg(p->db)); + rc = 1; + } + sqlite3_close(pSrc); + }else + + if( c=='s' && strncmp(azArg[0], "schema", n)==0 && nArg<3 ){ + struct callback_data data; + char *zErrMsg = 0; + open_db(p); + memcpy(&data, p, sizeof(data)); + data.showHeader = 0; + data.mode = MODE_Semi; + if( nArg>1 ){ + int i; + for(i=0; azArg[1][i]; i++) azArg[1][i] = (char)tolower(azArg[1][i]); + if( strcmp(azArg[1],"sqlite_master")==0 ){ + char *new_argv[2], *new_colv[2]; + new_argv[0] = "CREATE TABLE sqlite_master (\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")"; + new_argv[1] = 0; + new_colv[0] = "sql"; + new_colv[1] = 0; + callback(&data, 1, new_argv, new_colv); + rc = SQLITE_OK; + }else if( strcmp(azArg[1],"sqlite_temp_master")==0 ){ + char *new_argv[2], *new_colv[2]; + new_argv[0] = "CREATE TEMP TABLE sqlite_temp_master (\n" + " type text,\n" + " name text,\n" + " tbl_name text,\n" + " rootpage integer,\n" + " sql text\n" + ")"; + new_argv[1] = 0; + new_colv[0] = "sql"; + new_colv[1] = 0; + callback(&data, 1, new_argv, new_colv); + rc = SQLITE_OK; + }else{ + zShellStatic = azArg[1]; + rc = sqlite3_exec(p->db, + "SELECT sql FROM " + " (SELECT sql sql, type type, tbl_name tbl_name, name name" + " FROM sqlite_master UNION ALL" + " SELECT sql, type, tbl_name, name FROM sqlite_temp_master) " + "WHERE tbl_name LIKE shellstatic() AND type!='meta' AND sql NOTNULL " + "ORDER BY substr(type,2,1), name", + callback, &data, &zErrMsg); + zShellStatic = 0; + } + }else{ + rc = sqlite3_exec(p->db, + "SELECT sql FROM " + " (SELECT sql sql, type type, tbl_name tbl_name, name name" + " FROM sqlite_master UNION ALL" + " SELECT sql, type, tbl_name, name FROM sqlite_temp_master) " + "WHERE type!='meta' AND sql NOTNULL AND name NOT LIKE 'sqlite_%'" + "ORDER BY substr(type,2,1), name", + callback, &data, &zErrMsg + ); + } + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + rc = 1; + }else if( rc != SQLITE_OK ){ + fprintf(stderr,"Error: querying schema information\n"); + rc = 1; + }else{ + rc = 0; + } + }else + + if( c=='s' && strncmp(azArg[0], "separator", n)==0 && nArg==2 ){ + sqlite3_snprintf(sizeof(p->separator), p->separator, + "%.*s", (int)sizeof(p->separator)-1, azArg[1]); + }else + + if( c=='s' && strncmp(azArg[0], "show", n)==0 && nArg==1 ){ + int i; + fprintf(p->out,"%9.9s: %s\n","echo", p->echoOn ? "on" : "off"); + fprintf(p->out,"%9.9s: %s\n","explain", p->explainPrev.valid ? "on" :"off"); + fprintf(p->out,"%9.9s: %s\n","headers", p->showHeader ? "on" : "off"); + fprintf(p->out,"%9.9s: %s\n","mode", modeDescr[p->mode]); + fprintf(p->out,"%9.9s: ", "nullvalue"); + output_c_string(p->out, p->nullvalue); + fprintf(p->out, "\n"); + fprintf(p->out,"%9.9s: %s\n","output", + strlen30(p->outfile) ? p->outfile : "stdout"); + fprintf(p->out,"%9.9s: ", "separator"); + output_c_string(p->out, p->separator); + fprintf(p->out, "\n"); + fprintf(p->out,"%9.9s: %s\n","stats", p->statsOn ? "on" : "off"); + fprintf(p->out,"%9.9s: ","width"); + for (i=0;i<(int)ArraySize(p->colWidth) && p->colWidth[i] != 0;i++) { + fprintf(p->out,"%d ",p->colWidth[i]); + } + fprintf(p->out,"\n"); + }else + + if( c=='s' && strncmp(azArg[0], "stats", n)==0 && nArg>1 && nArg<3 ){ + p->statsOn = booleanValue(azArg[1]); + }else + + if( c=='t' && n>1 && strncmp(azArg[0], "tables", n)==0 && nArg<3 ){ + char **azResult; + int nRow; + char *zErrMsg; + open_db(p); + if( nArg==1 ){ + rc = sqlite3_get_table(p->db, + "SELECT name FROM sqlite_master " + "WHERE type IN ('table','view') AND name NOT LIKE 'sqlite_%' " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type IN ('table','view') " + "ORDER BY 1", + &azResult, &nRow, 0, &zErrMsg + ); + }else{ + zShellStatic = azArg[1]; + rc = sqlite3_get_table(p->db, + "SELECT name FROM sqlite_master " + "WHERE type IN ('table','view') AND name LIKE shellstatic() " + "UNION ALL " + "SELECT name FROM sqlite_temp_master " + "WHERE type IN ('table','view') AND name LIKE shellstatic() " + "ORDER BY 1", + &azResult, &nRow, 0, &zErrMsg + ); + zShellStatic = 0; + } + if( zErrMsg ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + sqlite3_free(zErrMsg); + rc = 1; + }else if( rc != SQLITE_OK ){ + fprintf(stderr,"Error: querying sqlite_master and sqlite_temp_master\n"); + rc = 1; + }else{ + int len, maxlen = 0; + int i, j; + int nPrintCol, nPrintRow; + for(i=1; i<=nRow; i++){ + if( azResult[i]==0 ) continue; + len = strlen30(azResult[i]); + if( len>maxlen ) maxlen = len; + } + nPrintCol = 80/(maxlen+2); + if( nPrintCol<1 ) nPrintCol = 1; + nPrintRow = (nRow + nPrintCol - 1)/nPrintCol; + for(i=0; i4 && strncmp(azArg[0], "timeout", n)==0 && nArg==2 ){ + open_db(p); + sqlite3_busy_timeout(p->db, atoi(azArg[1])); + }else + + if( HAS_TIMER && c=='t' && n>=5 && strncmp(azArg[0], "timer", n)==0 && nArg==2 ){ + enableTimer = booleanValue(azArg[1]); + }else + + if( c=='w' && strncmp(azArg[0], "width", n)==0 && nArg>1 ){ + int j; + assert( nArg<=ArraySize(azArg) ); + for(j=1; jcolWidth); j++){ + p->colWidth[j-1] = atoi(azArg[j]); + } + }else + + { + fprintf(stderr, "Error: unknown command or invalid arguments: " + " \"%s\". Enter \".help\" for help\n", azArg[0]); + rc = 1; + } + + return rc; +} + +/* +** Return TRUE if a semicolon occurs anywhere in the first N characters +** of string z[]. +*/ +static int _contains_semicolon(const char *z, int N){ + int i; + for(i=0; iout); + free(zLine); + zLine = one_input_line(zSql, in); + if( zLine==0 ){ + break; /* We have reached EOF */ + } + if( seenInterrupt ){ + if( in!=0 ) break; + seenInterrupt = 0; + } + lineno++; + if( (zSql==0 || zSql[0]==0) && _all_whitespace(zLine) ) continue; + if( zLine && zLine[0]=='.' && nSql==0 ){ + if( p->echoOn ) printf("%s\n", zLine); + rc = do_meta_command(zLine, p); + if( rc==2 ){ /* exit requested */ + break; + }else if( rc ){ + errCnt++; + } + continue; + } + if( _is_command_terminator(zLine) && _is_complete(zSql, nSql) ){ + memcpy(zLine,";",2); + } + nSqlPrior = nSql; + if( zSql==0 ){ + int i; + for(i=0; zLine[i] && isspace((unsigned char)zLine[i]); i++){} + if( zLine[i]!=0 ){ + nSql = strlen30(zLine); + zSql = malloc( nSql+3 ); + if( zSql==0 ){ + fprintf(stderr, "Error: out of memory\n"); + exit(1); + } + memcpy(zSql, zLine, nSql+1); + startline = lineno; + } + }else{ + int len = strlen30(zLine); + zSql = realloc( zSql, nSql + len + 4 ); + if( zSql==0 ){ + fprintf(stderr,"Error: out of memory\n"); + exit(1); + } + zSql[nSql++] = '\n'; + memcpy(&zSql[nSql], zLine, len+1); + nSql += len; + } + if( zSql && _contains_semicolon(&zSql[nSqlPrior], nSql-nSqlPrior) + && sqlite3_complete(zSql) ){ + p->cnt = 0; + open_db(p); + BEGIN_TIMER; + rc = shell_exec(p->db, zSql, shell_callback, p, &zErrMsg); + END_TIMER; + if( rc || zErrMsg ){ + char zPrefix[100]; + if( in!=0 || !stdin_is_interactive ){ + sqlite3_snprintf(sizeof(zPrefix), zPrefix, + "Error: near line %d:", startline); + }else{ + sqlite3_snprintf(sizeof(zPrefix), zPrefix, "Error:"); + } + if( zErrMsg!=0 ){ + fprintf(stderr, "%s %s\n", zPrefix, zErrMsg); + sqlite3_free(zErrMsg); + zErrMsg = 0; + }else{ + fprintf(stderr, "%s %s\n", zPrefix, sqlite3_errmsg(p->db)); + } + errCnt++; + } + free(zSql); + zSql = 0; + nSql = 0; + } + } + if( zSql ){ + if( !_all_whitespace(zSql) ) fprintf(stderr, "Error: incomplete SQL: %s\n", zSql); + free(zSql); + } + free(zLine); + return errCnt; +} + +/* +** Return a pathname which is the user's home directory. A +** 0 return indicates an error of some kind. Space to hold the +** resulting string is obtained from malloc(). The calling +** function should free the result. +*/ +static char *find_home_dir(void){ + char *home_dir = NULL; + +#if !defined(_WIN32) && !defined(WIN32) && !defined(__OS2__) && !defined(_WIN32_WCE) && !defined(__RTP__) && !defined(_WRS_KERNEL) + struct passwd *pwent; + uid_t uid = getuid(); + if( (pwent=getpwuid(uid)) != NULL) { + home_dir = pwent->pw_dir; + } +#endif + +#if defined(_WIN32_WCE) + /* Windows CE (arm-wince-mingw32ce-gcc) does not provide getenv() + */ + home_dir = strdup("/"); +#else + +#if defined(_WIN32) || defined(WIN32) || defined(__OS2__) + if (!home_dir) { + home_dir = getenv("USERPROFILE"); + } +#endif + + if (!home_dir) { + home_dir = getenv("HOME"); + } + +#if defined(_WIN32) || defined(WIN32) || defined(__OS2__) + if (!home_dir) { + char *zDrive, *zPath; + int n; + zDrive = getenv("HOMEDRIVE"); + zPath = getenv("HOMEPATH"); + if( zDrive && zPath ){ + n = strlen30(zDrive) + strlen30(zPath) + 1; + home_dir = malloc( n ); + if( home_dir==0 ) return 0; + sqlite3_snprintf(n, home_dir, "%s%s", zDrive, zPath); + return home_dir; + } + home_dir = "c:\\"; + } +#endif + +#endif /* !_WIN32_WCE */ + + if( home_dir ){ + int n = strlen30(home_dir) + 1; + char *z = malloc( n ); + if( z ) memcpy(z, home_dir, n); + home_dir = z; + } + + return home_dir; +} + +/* +** Read input from the file given by sqliterc_override. Or if that +** parameter is NULL, take input from ~/.sqliterc +** +** Returns the number of errors. +*/ +static int process_sqliterc( + struct callback_data *p, /* Configuration data */ + const char *sqliterc_override /* Name of config file. NULL to use default */ +){ + char *home_dir = NULL; + const char *sqliterc = sqliterc_override; + char *zBuf = 0; + FILE *in = NULL; + int nBuf; + int rc = 0; + + if (sqliterc == NULL) { + home_dir = find_home_dir(); + if( home_dir==0 ){ +#if !defined(__RTP__) && !defined(_WRS_KERNEL) + fprintf(stderr,"%s: Error: cannot locate your home directory\n", Argv0); +#endif + return 1; + } + nBuf = strlen30(home_dir) + 16; + zBuf = malloc( nBuf ); + if( zBuf==0 ){ + fprintf(stderr,"%s: Error: out of memory\n",Argv0); + return 1; + } + sqlite3_snprintf(nBuf, zBuf,"%s/.sqliterc",home_dir); + free(home_dir); + sqliterc = (const char*)zBuf; + } + in = fopen(sqliterc,"rb"); + if( in ){ + if( stdin_is_interactive ){ + fprintf(stderr,"-- Loading resources from %s\n",sqliterc); + } + rc = process_input(p,in); + fclose(in); + } + free(zBuf); + return rc; +} + +/* +** Show available command line options +*/ +static const char zOptions[] = + " -help show this message\n" + " -init filename read/process named file\n" + " -echo print commands before execution\n" + " -[no]header turn headers on or off\n" + " -bail stop after hitting an error\n" + " -interactive force interactive I/O\n" + " -batch force batch I/O\n" + " -column set output mode to 'column'\n" + " -csv set output mode to 'csv'\n" + " -html set output mode to HTML\n" + " -line set output mode to 'line'\n" + " -list set output mode to 'list'\n" + " -separator 'x' set output field separator (|)\n" + " -stats print memory stats before each finalize\n" + " -nullvalue 'text' set text string for NULL values\n" + " -version show SQLite version\n" +; +static void usage(int showDetail){ + fprintf(stderr, + "Usage: %s [OPTIONS] FILENAME [SQL]\n" + "FILENAME is the name of an SQLite database. A new database is created\n" + "if the file does not previously exist.\n", Argv0); + if( showDetail ){ + fprintf(stderr, "OPTIONS include:\n%s", zOptions); + }else{ + fprintf(stderr, "Use the -help option for additional information\n"); + } + exit(1); +} + +/* +** Initialize the state information in data +*/ +static void main_init(struct callback_data *data) { + memset(data, 0, sizeof(*data)); + data->mode = MODE_List; + memcpy(data->separator,"|", 2); + data->showHeader = 0; + sqlite3_config(SQLITE_CONFIG_LOG, shellLog, data); + sqlite3_snprintf(sizeof(mainPrompt), mainPrompt,"sqlite> "); + sqlite3_snprintf(sizeof(continuePrompt), continuePrompt," ...> "); + sqlite3_config(SQLITE_CONFIG_SINGLETHREAD); +} + +int main(int argc, char **argv){ + char *zErrMsg = 0; + struct callback_data data; + const char *zInitFile = 0; + char *zFirstCmd = 0; + int i; + int rc = 0; + + Argv0 = argv[0]; + main_init(&data); + stdin_is_interactive = isatty(0); + + /* Make sure we have a valid signal handler early, before anything + ** else is done. + */ +#ifdef SIGINT + signal(SIGINT, interrupt_handler); +#endif + + /* Do an initial pass through the command-line argument to locate + ** the name of the database file, the name of the initialization file, + ** and the first command to execute. + */ + for(i=1; i0 ){ + return rc; + } + + /* Make a second pass through the command-line argument and set + ** options. This second pass is delayed until after the initialization + ** file is processed so that the command-line arguments will override + ** settings in the initialization file. + */ + for(i=1; i=argc){ + fprintf(stderr,"%s: Error: missing argument for option: %s\n", Argv0, z); + fprintf(stderr,"Use -help for a list of options.\n"); + return 1; + } + sqlite3_snprintf(sizeof(data.separator), data.separator, + "%.*s",(int)sizeof(data.separator)-1,argv[i]); + }else if( strcmp(z,"-nullvalue")==0 ){ + i++; + if(i>=argc){ + fprintf(stderr,"%s: Error: missing argument for option: %s\n", Argv0, z); + fprintf(stderr,"Use -help for a list of options.\n"); + return 1; + } + sqlite3_snprintf(sizeof(data.nullvalue), data.nullvalue, + "%.*s",(int)sizeof(data.nullvalue)-1,argv[i]); + }else if( strcmp(z,"-header")==0 ){ + data.showHeader = 1; + }else if( strcmp(z,"-noheader")==0 ){ + data.showHeader = 0; + }else if( strcmp(z,"-echo")==0 ){ + data.echoOn = 1; + }else if( strcmp(z,"-stats")==0 ){ + data.statsOn = 1; + }else if( strcmp(z,"-bail")==0 ){ + bail_on_error = 1; + }else if( strcmp(z,"-version")==0 ){ + printf("%s\n", sqlite3_libversion()); + return 0; + }else if( strcmp(z,"-interactive")==0 ){ + stdin_is_interactive = 1; + }else if( strcmp(z,"-batch")==0 ){ + stdin_is_interactive = 0; + }else if( strcmp(z,"-help")==0 || strcmp(z, "--help")==0 ){ + usage(1); + }else{ + fprintf(stderr,"%s: Error: unknown option: %s\n", Argv0, z); + fprintf(stderr,"Use -help for a list of options.\n"); + return 1; + } + } + + if( zFirstCmd ){ + /* Run just the command that follows the database name + */ + if( zFirstCmd[0]=='.' ){ + rc = do_meta_command(zFirstCmd, &data); + }else{ + open_db(&data); + rc = shell_exec(data.db, zFirstCmd, shell_callback, &data, &zErrMsg); + if( zErrMsg!=0 ){ + fprintf(stderr,"Error: %s\n", zErrMsg); + return rc!=0 ? rc : 1; + }else if( rc!=0 ){ + fprintf(stderr,"Error: unable to process SQL \"%s\"\n", zFirstCmd); + return rc; + } + } + }else{ + /* Run commands received from standard input + */ + if( stdin_is_interactive ){ + char *zHome; + char *zHistory = 0; + int nHistory; + printf( + "SQLite version %s\n" + "Enter \".help\" for instructions\n" + "Enter SQL statements terminated with a \";\"\n", + sqlite3_libversion() + ); + zHome = find_home_dir(); + if( zHome ){ + nHistory = strlen30(zHome) + 20; + if( (zHistory = malloc(nHistory))!=0 ){ + sqlite3_snprintf(nHistory, zHistory,"%s/.sqlite_history", zHome); + } + } +#if defined(HAVE_READLINE) && HAVE_READLINE==1 + if( zHistory ) read_history(zHistory); +#endif + rc = process_input(&data, 0); + if( zHistory ){ + stifle_history(100); + write_history(zHistory); + free(zHistory); + } + free(zHome); + }else{ + rc = process_input(&data, stdin); + } + } + set_table_name(&data, 0); + if( data.db ){ + if( sqlite3_close(data.db)!=SQLITE_OK ){ + fprintf(stderr,"Error: cannot close database \"%s\"\n", + sqlite3_errmsg(db)); + rc++; + } + } + return rc; +} Index: src/shun.c ================================================================== --- src/shun.c +++ src/shun.c @@ -69,29 +69,29 @@ style_header("Shunned Artifacts"); if( zUuid && P("sub") ){ login_verify_csrf_secret(); db_multi_exec("DELETE FROM shun WHERE uuid='%s'", zUuid); if( db_exists("SELECT 1 FROM blob WHERE uuid='%s'", zUuid) ){ - @

Artifact + @

Artifact @ %s(zUuid) is no - @ longer being shunned.

+ @ longer being shunned.

}else{ - @

Artifact %s(zUuid) will no longer + @

Artifact %s(zUuid) will no longer @ be shunned. But it does not exist in the repository. It @ may be necessary to rebuild the repository using the @ fossil rebuild command-line before the artifact content - @ can pulled in from other respositories.

+ @ can pulled in from other respositories.

} } if( zUuid && P("add") ){ login_verify_csrf_secret(); db_multi_exec("INSERT OR IGNORE INTO shun VALUES('%s')", zUuid); - @

Artifact + @

Artifact @ %s(zUuid) has been @ shunned. It will no longer be pushed. @ It will be removed from the repository the next time the respository - @ is rebuilt using the fossil rebuild command-line

+ @ is rebuilt using the fossil rebuild command-line

} @

A shunned artifact will not be pushed nor accepted in a pull and the @ artifact content will be purged from the repository the next time the @ repository is rebuilt. A list of shunned artifacts can be seen at the @ bottom of this page.

@@ -112,63 +112,63 @@ @ or artifacts that by design or accident interfere with the processing @ of the repository. Do not shun artifacts merely to remove them from @ sight - set the "hidden" tag on such artifacts instead.

@ @

- @
+ @
login_insert_csrf_secret(); - @ - @ - @ + @ + @ + @
@

@ @

Enter the UUID of a previous shunned artifact to cause it to be @ accepted again in the repository. The artifact content is not @ restored because the content is unknown. The only change is that @ the formerly shunned artifact will be accepted on subsequent sync @ operations.

@ @

- @
+ @
login_insert_csrf_secret(); - @ - @ - @ + @ + @ + @
@

@ @

Press the Rebuild button below to rebuild the respository. The @ content of newly shunned artifacts is not purged until the repository @ is rebuilt. On larger repositories, the rebuild may take minute or @ two, so be patient after pressing the button.

@ @

- @
+ @
login_insert_csrf_secret(); - @ - @ + @ + @
@

@ - @

Shunned Artifacts:

- @

+ @
Shunned Artifacts:
+ @
db_prepare(&q, "SELECT uuid, EXISTS(SELECT 1 FROM blob WHERE blob.uuid=shun.uuid)" " FROM shun ORDER BY uuid"); while( db_step(&q)==SQLITE_ROW ){ const char *zUuid = db_column_text(&q, 0); int stillExists = db_column_int(&q, 1); cnt++; if( stillExists ){ - @ %s(zUuid)
+ @ %s(zUuid)
}else{ - @ %s(zUuid)
+ @ %s(zUuid)
} } if( cnt==0 ){ @ no artifacts are shunned on this server } db_finalize(&q); - @
+ @

style_footer(); } /* ** Remove from the BLOB table all artifacts that are in the SHUN table. @@ -229,14 +229,15 @@ @ @

Click on the "rcvid" to show a list of specific artifacts received @ by a transaction. After identifying illicit artifacts, remove them @ using the "Shun" feature.

@ - @

- @ - @ + @

rcvid	- @	Date		User		IP Address

+ @ + @ + @ + @ cnt = 0; while( db_step(&q)==SQLITE_ROW ){ int rcvid = db_column_int(&q, 0); const char *zUser = db_column_text(&q, 1); const char *zDate = db_column_text(&q, 2); @@ -245,14 +246,14 @@ style_submenu_element("Older", "Older", "rcvfromlist?ofst=%d", ofst+30); }else{ cnt++; @ - @ + @ + @ + @ + @ @ } } db_finalize(&q); @

rcvid	Date	User	IP Address
%d(rcvid)	- @	%s(zDate)	- @	%h(zUser)	- @	%s(zIpAddr)	%d(rcvid)	%s(zDate)	%h(zUser)	%s(zIpAddr)

@@ -277,11 +278,11 @@ "SELECT login, datetime(rcvfrom.mtime), rcvfrom.ipaddr" " FROM rcvfrom LEFT JOIN user USING(uid)" " WHERE rcvid=%d", rcvid ); - @ + @

@ @ if( db_step(&q)==SQLITE_ROW ){ const char *zUser = db_column_text(&q, 0); const char *zDate = db_column_text(&q, 1); @@ -302,10 +303,12 @@ while( db_step(&q)==SQLITE_ROW ){ int rid = db_column_int(&q, 0); const char *zUuid = db_column_text(&q, 1); int size = db_column_int(&q, 2); @ %s(zUuid) - @ (rid: %d(rid), size: %d(size))
+ @ (rid: %d(rid), size: %d(size))
} @ @

rcvid:

%d(rcvid)

+ db_finalize(&q); + style_footer(); } Index: src/skins.c ================================================================== --- src/skins.c +++ src/skins.c @@ -165,11 +165,11 @@ @ @

$
$</div> +@ <div class="title"><small>$<project_name></small><br />$<title></div> @ <div class="status"><nobr><th1> @ if {[info exists login]} { @ puts "Logged in as $login" @ } else { @ puts "Not logged in" @@ -600,11 +600,11 @@ @ </head> @ <body> @ <div class="header"> @ <div class="logo"> @  -@ <br><nobr>$<project_name></nobr> +@ <br /><nobr>$<project_name></nobr> @ </div> @ <div class="title">$<title></div> @ <div class="status"><nobr><th1> @ if {[info exists login]} { @ puts "Logged in as $login" @@ -656,15 +656,16 @@ ** An array of available built-in skins. */ static struct BuiltinSkin { const char *zName; const char *zValue; + const char *zAuthor; } aBuiltinSkin[] = { - { "Default", 0 /* Filled in at runtime */ }, - { "Plain Gray, No Logo", zBuiltinSkin1 }, - { "Khaki, No Logo", zBuiltinSkin2 }, - { "Black & White, Menu on Left", zBuiltinSkin3 }, + { "Default", 0 /* Filled in at runtime */, 0}, + { "Plain Gray, No Logo", zBuiltinSkin1, "Unknown"}, + { "Khaki, No Logo", zBuiltinSkin2, "Unknown"}, + { "Black & White, Menu on Left", zBuiltinSkin3, "Unknown"}, }; /* ** For a skin named zSkinName, compute the name of the CONFIG table ** entry where that skin is stored and return it. @@ -731,18 +732,18 @@ db_begin_transaction(); /* Process requests to delete a user-defined skin */ if( P("del1") && (zName = skinVarName(P("sn"), 1))!=0 ){ style_header("Confirm Custom Skin Delete"); - @ <form action="%s(g.zBaseURL)/setup_skin" method="POST"> + @ <form action="%s(g.zBaseURL)/setup_skin" method="post"><div> @ <p>Deletion of a custom skin is a permanent action that cannot @ be undone. Please confirm that this is what you want to do:</p> - @ <input type="hidden" name="sn" value="%h(P("sn"))"> - @ <input type="submit" name="del2" value="Confirm - Delete The Skin"> - @ <input type="submit" name="cancel" value="Cancel - Do Not Delete"> + @ <input type="hidden" name="sn" value="%h(P("sn"))" /> + @ <input type="submit" name="del2" value="Confirm - Delete The Skin" /> + @ <input type="submit" name="cancel" value="Cancel - Do Not Delete" /> login_insert_csrf_secret(); - @ </form> + @ </div></form> style_footer(); return; } if( P("del2")!=0 && (zName = skinVarName(P("sn"), 1))!=0 ){ db_multi_exec("DELETE FROM config WHERE name=%Q", zName); @@ -812,15 +813,15 @@ for(i=0; i<sizeof(aBuiltinSkin)/sizeof(aBuiltinSkin[0]); i++){ z = aBuiltinSkin[i].zName; if( strcmp(aBuiltinSkin[i].zValue, zCurrent)==0 ){ @ <li><p>%h(z). <b>Currently In Use</b></p> }else{ - @ <li><form action="%s(g.zBaseURL)/setup_skin" method="POST"> + @ <li><form action="%s(g.zBaseURL)/setup_skin" method="post"><div> @ %h(z). - @ <input type="hidden" name="sn" value="%h(z)"> - @ <input type="submit" name="load" value="Use This Skin"> - @ </form></li> + @ <input type="hidden" name="sn" value="%h(z)" /> + @ <input type="submit" name="load" value="Use This Skin" /> + @ </div></form></li> } } db_prepare(&q, "SELECT substr(name, 6), value FROM config" " WHERE name GLOB 'skin:*'" @@ -830,11 +831,11 @@ const char *zN = db_column_text(&q, 0); const char *zV = db_column_text(&q, 1); if( strcmp(zV, zCurrent)==0 ){ @ <li><p>%h(zN). <b>Currently In Use</b></p> }else{ - @ <li><form action="%s(g.zBaseURL)/setup_skin" method="POST"> + @ <li><form action="%s(g.zBaseURL)/setup_skin" method="post"> @ %h(zN). @ <input type="hidden" name="sn" value="%h(zN)"> @ <input type="submit" name="load" value="Use This Skin"> @ <input type="submit" name="del1" value="Delete This Skin"> @ </form></li> Index: src/sqlite3.c ================================================================== --- src/sqlite3.c +++ src/sqlite3.c @@ -1,8 +1,8 @@ /****************************************************************************** ** This file is an amalgamation of many separate C source files from SQLite -** version 3.7.1. By combining all the individual C code files into this +** version 3.7.3. By combining all the individual C code files into this ** single large file, the entire code can be compiled as a one translation ** unit. This allows many compilers to do optimizations that would not be ** possible if the files were compiled separately. Performance improvements ** of 5% are more are commonly seen when SQLite is compiled as a single ** translation unit. @@ -213,24 +213,25 @@ */ #ifndef SQLITE_MAX_VARIABLE_NUMBER # define SQLITE_MAX_VARIABLE_NUMBER 999 #endif -/* Maximum page size. The upper bound on this value is 32768. This a limit -** imposed by the necessity of storing the value in a 2-byte unsigned integer -** and the fact that the page size must be a power of 2. +/* Maximum page size. The upper bound on this value is 65536. This a limit +** imposed by the use of 16-bit offsets within each page. ** -** If this limit is changed, then the compiled library is technically -** incompatible with an SQLite library compiled with a different limit. If -** a process operating on a database with a page-size of 65536 bytes -** crashes, then an instance of SQLite compiled with the default page-size -** limit will not be able to rollback the aborted transaction. This could -** lead to database corruption. +** Earlier versions of SQLite allowed the user to change this value at +** compile time. This is no longer permitted, on the grounds that it creates +** a library that is technically incompatible with an SQLite library +** compiled with a different limit. If a process operating on a database +** with a page-size of 65536 bytes crashes, then an instance of SQLite +** compiled with the default page-size limit will not be able to rollback +** the aborted transaction. This could lead to database corruption. */ -#ifndef SQLITE_MAX_PAGE_SIZE -# define SQLITE_MAX_PAGE_SIZE 32768 +#ifdef SQLITE_MAX_PAGE_SIZE +# undef SQLITE_MAX_PAGE_SIZE #endif +#define SQLITE_MAX_PAGE_SIZE 65536 /* ** The default size of a database page. */ @@ -351,19 +352,25 @@ # define SQLITE_INT_TO_PTR(X) ((void*)(X)) # define SQLITE_PTR_TO_INT(X) ((int)(X)) #endif /* -** The SQLITE_THREADSAFE macro must be defined as either 0 or 1. +** The SQLITE_THREADSAFE macro must be defined as 0, 1, or 2. +** 0 means mutexes are permanently disable and the library is never +** threadsafe. 1 means the library is serialized which is the highest +** level of threadsafety. 2 means the libary is multithreaded - multiple +** threads can use SQLite as long as no two threads try to use the same +** database connection at the same time. +** ** Older versions of SQLite used an optional THREADSAFE macro. -** We support that for legacy +** We support that for legacy. */ #if !defined(SQLITE_THREADSAFE) #if defined(THREADSAFE) # define SQLITE_THREADSAFE THREADSAFE #else -# define SQLITE_THREADSAFE 1 +# define SQLITE_THREADSAFE 1 /* IMP: R-07272-22309 */ #endif #endif /* ** The SQLITE_DEFAULT_MEMSTATUS macro must be defined as either 0 or 1. @@ -631,23 +638,23 @@ ** be held constant and Z will be incremented or else Y will be incremented ** and Z will be reset to zero. ** ** Since version 3.6.18, SQLite source code has been stored in the ** <a href="http://www.fossil-scm.org/">Fossil configuration management -** system</a>. ^The SQLITE_SOURCE_ID macro evalutes to +** system</a>. ^The SQLITE_SOURCE_ID macro evaluates to ** a string which identifies a particular check-in of SQLite ** within its configuration management system. ^The SQLITE_SOURCE_ID ** string contains the date and time of the check-in (UTC) and an SHA1 ** hash of the entire source tree. ** ** See also: [sqlite3_libversion()], ** [sqlite3_libversion_number()], [sqlite3_sourceid()], ** [sqlite_version()] and [sqlite_source_id()]. */ -#define SQLITE_VERSION "3.7.1" -#define SQLITE_VERSION_NUMBER 3007001 -#define SQLITE_SOURCE_ID "2010-08-05 03:21:40 fbe70e1106bcc5086ceb9d8f39cc39baf3643092" +#define SQLITE_VERSION "3.7.3" +#define SQLITE_VERSION_NUMBER 3007003 +#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e" /* ** CAPI3REF: Run-Time Library Version Numbers ** KEYWORDS: sqlite3_version, sqlite3_sourceid ** @@ -688,19 +695,19 @@ ** ^The sqlite3_compileoption_used() function returns 0 or 1 ** indicating whether the specified option was defined at ** compile time. ^The SQLITE_ prefix may be omitted from the ** option name passed to sqlite3_compileoption_used(). ** -** ^The sqlite3_compileoption_get() function allows interating +** ^The sqlite3_compileoption_get() function allows iterating ** over the list of options that were defined at compile time by ** returning the N-th compile time option string. ^If N is out of range, ** sqlite3_compileoption_get() returns a NULL pointer. ^The SQLITE_ ** prefix is omitted from any strings returned by ** sqlite3_compileoption_get(). ** ** ^Support for the diagnostic functions sqlite3_compileoption_used() -** and sqlite3_compileoption_get() may be omitted by specifing the +** and sqlite3_compileoption_get() may be omitted by specifying the ** [SQLITE_OMIT_COMPILEOPTION_DIAGS] option at compile time. ** ** See also: SQL functions [sqlite_compileoption_used()] and ** [sqlite_compileoption_get()] and the [compile_options pragma]. */ @@ -802,11 +809,11 @@ /* ** CAPI3REF: Closing A Database Connection ** ** ^The sqlite3_close() routine is the destructor for the [sqlite3] object. ** ^Calls to sqlite3_close() return SQLITE_OK if the [sqlite3] object is -** successfullly destroyed and all associated resources are deallocated. +** successfully destroyed and all associated resources are deallocated. ** ** Applications must [sqlite3_finalize | finalize] all [prepared statements] ** and [sqlite3_blob_close | close] all [BLOB handles] associated with ** the [sqlite3] object prior to attempting to close the object. ^If ** sqlite3_close() is called on a [database connection] that still has @@ -1229,16 +1236,25 @@ ** layer a hint of how large the database file will grow to be during the ** current transaction. This hint is not guaranteed to be accurate but it ** is often close. The underlying VFS might choose to preallocate database ** file space based on this hint in order to help writes to the database ** file run faster. +** +** The [SQLITE_FCNTL_CHUNK_SIZE] opcode is used to request that the VFS +** extends and truncates the database file in chunks of a size specified +** by the user. The fourth argument to [sqlite3_file_control()] should +** point to an integer (type int) containing the new chunk-size to use +** for the nominated database. Allocating database file space in large +** chunks (say 1MB at a time), may reduce file-system fragmentation and +** improve performance on some systems. */ #define SQLITE_FCNTL_LOCKSTATE 1 #define SQLITE_GET_LOCKPROXYFILE 2 #define SQLITE_SET_LOCKPROXYFILE 3 #define SQLITE_LAST_ERRNO 4 #define SQLITE_FCNTL_SIZE_HINT 5 +#define SQLITE_FCNTL_CHUNK_SIZE 6 /* ** CAPI3REF: Mutex Handle ** ** The mutex module within SQLite defines [sqlite3_mutex] to be an @@ -1282,19 +1298,23 @@ ** object once the object has been registered. ** ** The zName field holds the name of the VFS module. The name must ** be unique across all VFS modules. ** -** SQLite will guarantee that the zFilename parameter to xOpen +** ^SQLite guarantees that the zFilename parameter to xOpen ** is either a NULL pointer or string obtained -** from xFullPathname(). SQLite further guarantees that +** from xFullPathname() with an optional suffix added. +** ^If a suffix is added to the zFilename parameter, it will +** consist of a single "-" character followed by no more than +** 10 alphanumeric and/or "-" characters. +** ^SQLite further guarantees that ** the string will be valid and unchanged until xClose() is ** called. Because of the previous sentence, ** the [sqlite3_file] can safely store a pointer to the ** filename if it needs to remember the filename for some reason. -** If the zFilename parameter is xOpen is a NULL pointer then xOpen -** must invent its own temporary name for the file. Whenever the +** If the zFilename parameter to xOpen is a NULL pointer then xOpen +** must invent its own temporary name for the file. ^Whenever the ** xFilename parameter is NULL it will also be the case that the ** flags parameter will include [SQLITE_OPEN_DELETEONCLOSE]. ** ** The flags argument to xOpen() includes all bits set in ** the flags argument to [sqlite3_open_v2()]. Or if [sqlite3_open()] @@ -1301,11 +1321,11 @@ ** or [sqlite3_open16()] is used, then flags includes at least ** [SQLITE_OPEN_READWRITE] | [SQLITE_OPEN_CREATE]. ** If xOpen() opens a file read-only then it sets *pOutFlags to ** include [SQLITE_OPEN_READONLY]. Other bits in *pOutFlags may be set. ** -** SQLite will also add one of the following flags to the xOpen() +** ^(SQLite will also add one of the following flags to the xOpen() ** call, depending on the object being opened: ** ** <ul> ** <li> [SQLITE_OPEN_MAIN_DB] ** <li> [SQLITE_OPEN_MAIN_JOURNAL] @@ -1312,11 +1332,12 @@ ** <li> [SQLITE_OPEN_TEMP_DB] ** <li> [SQLITE_OPEN_TEMP_JOURNAL] ** <li> [SQLITE_OPEN_TRANSIENT_DB] ** <li> [SQLITE_OPEN_SUBJOURNAL] ** <li> [SQLITE_OPEN_MASTER_JOURNAL] -** </ul> +** <li> [SQLITE_OPEN_WAL] +** </ul>)^ ** ** The file I/O implementation can use the object type flags to ** change the way it deals with files. For example, an application ** that does not care about crash recovery or rollback might make ** the open of a journal file a no-op. Writes to this journal would @@ -1331,39 +1352,40 @@ ** <li> [SQLITE_OPEN_DELETEONCLOSE] ** <li> [SQLITE_OPEN_EXCLUSIVE] ** </ul> ** ** The [SQLITE_OPEN_DELETEONCLOSE] flag means the file should be -** deleted when it is closed. The [SQLITE_OPEN_DELETEONCLOSE] -** will be set for TEMP databases, journals and for subjournals. +** deleted when it is closed. ^The [SQLITE_OPEN_DELETEONCLOSE] +** will be set for TEMP databases and their journals, transient +** databases, and subjournals. ** -** The [SQLITE_OPEN_EXCLUSIVE] flag is always used in conjunction +** ^The [SQLITE_OPEN_EXCLUSIVE] flag is always used in conjunction ** with the [SQLITE_OPEN_CREATE] flag, which are both directly ** analogous to the O_EXCL and O_CREAT flags of the POSIX open() ** API. The SQLITE_OPEN_EXCLUSIVE flag, when paired with the ** SQLITE_OPEN_CREATE, is used to indicate that file should always ** be created, and that it is an error if it already exists. ** It is <i>not</i> used to indicate the file should be opened ** for exclusive access. ** -** At least szOsFile bytes of memory are allocated by SQLite +** ^At least szOsFile bytes of memory are allocated by SQLite ** to hold the [sqlite3_file] structure passed as the third ** argument to xOpen. The xOpen method does not have to ** allocate the structure; it should just fill it in. Note that ** the xOpen method must set the sqlite3_file.pMethods to either ** a valid [sqlite3_io_methods] object or to NULL. xOpen must do ** this even if the open fails. SQLite expects that the sqlite3_file.pMethods ** element will be valid after xOpen returns regardless of the success ** or failure of the xOpen call. ** -** The flags argument to xAccess() may be [SQLITE_ACCESS_EXISTS] +** ^The flags argument to xAccess() may be [SQLITE_ACCESS_EXISTS] ** to test for the existence of a file, or [SQLITE_ACCESS_READWRITE] to ** test whether a file is readable and writable, or [SQLITE_ACCESS_READ] ** to test whether a file is at least readable. The file can be a ** directory. ** -** SQLite will always allocate at least mxPathname+1 bytes for the +** ^SQLite will always allocate at least mxPathname+1 bytes for the ** output buffer xFullPathname. The exact size of the output buffer ** is also passed as a parameter to both methods. If the output buffer ** is not large enough, [SQLITE_CANTOPEN] should be returned. Since this is ** handled as a fatal error by SQLite, vfs implementations should endeavor ** to prevent this by setting mxPathname to a sufficiently large value. @@ -1373,14 +1395,14 @@ ** included in the VFS structure for completeness. ** The xRandomness() function attempts to return nBytes bytes ** of good-quality randomness into zOut. The return value is ** the actual number of bytes of randomness obtained. ** The xSleep() method causes the calling thread to sleep for at -** least the number of microseconds given. The xCurrentTime() +** least the number of microseconds given. ^The xCurrentTime() ** method returns a Julian Day Number for the current date and time as ** a floating point value. -** The xCurrentTimeInt64() method returns, as an integer, the Julian +** ^The xCurrentTimeInt64() method returns, as an integer, the Julian ** Day Number multipled by 86400000 (the number of milliseconds in ** a 24-hour day). ** ^SQLite will use the xCurrentTimeInt64() method to get the current ** date and time if that method is available (if iVersion is 2 or ** greater and the function pointer is not NULL) and will fall back @@ -1773,11 +1795,11 @@ ** statistics. ^(When memory allocation statistics are disabled, the ** following SQLite interfaces become non-operational: ** <ul> ** <li> [sqlite3_memory_used()] ** <li> [sqlite3_memory_highwater()] -** <li> [sqlite3_soft_heap_limit()] +** <li> [sqlite3_soft_heap_limit64()] ** <li> [sqlite3_status()] ** </ul>)^ ** ^Memory allocation statistics are enabled by default unless SQLite is ** compiled with [SQLITE_DEFAULT_MEMSTATUS]=0 in which case memory ** allocation statistics are disabled by default. @@ -1787,19 +1809,18 @@ ** <dd> ^This option specifies a static memory buffer that SQLite can use for ** scratch memory. There are three arguments: A pointer an 8-byte ** aligned memory buffer from which the scrach allocations will be ** drawn, the size of each scratch allocation (sz), ** and the maximum number of scratch allocations (N). The sz -** argument must be a multiple of 16. The sz parameter should be a few bytes -** larger than the actual scratch space required due to internal overhead. +** argument must be a multiple of 16. ** The first argument must be a pointer to an 8-byte aligned buffer ** of at least sz*N bytes of memory. -** ^SQLite will use no more than one scratch buffer per thread. So -** N should be set to the expected maximum number of threads. ^SQLite will -** never require a scratch buffer that is more than 6 times the database -** page size. ^If SQLite needs needs additional scratch memory beyond -** what is provided by this configuration option, then +** ^SQLite will use no more than two scratch buffers per thread. So +** N should be set to twice the expected maximum number of threads. +** ^SQLite will never require a scratch buffer that is more than 6 +** times the database page size. ^If SQLite needs needs additional +** scratch memory beyond what is provided by this configuration option, then ** [sqlite3_malloc()] will be used to obtain the memory needed.</dd> ** ** <dt>SQLITE_CONFIG_PAGECACHE</dt> ** <dd> ^This option specifies a static memory buffer that SQLite can use for ** the database page cache with the default page cache implemenation. @@ -1815,12 +1836,11 @@ ** argument should point to an allocation of at least sz*N bytes of memory. ** ^SQLite will use the memory provided by the first argument to satisfy its ** memory needs for the first N pages that it adds to cache. ^If additional ** page cache memory is needed beyond what is provided by this option, then ** SQLite goes to [sqlite3_malloc()] for the additional storage space. -** ^The implementation might use one or more of the N buffers to hold -** memory accounting information. The pointer in the first argument must +** The pointer in the first argument must ** be aligned to an 8-byte boundary or subsequent behavior of SQLite ** will be undefined.</dd> ** ** <dt>SQLITE_CONFIG_HEAP</dt> ** <dd> ^This option specifies a static memory buffer that SQLite will use @@ -1945,12 +1965,18 @@ ** size of each lookaside buffer slot. ^The third argument is the number of ** slots. The size of the buffer in the first argument must be greater than ** or equal to the product of the second and third arguments. The buffer ** must be aligned to an 8-byte boundary. ^If the second argument to ** SQLITE_DBCONFIG_LOOKASIDE is not a multiple of 8, it is internally -** rounded down to the next smaller -** multiple of 8. See also: [SQLITE_CONFIG_LOOKASIDE]</dd> +** rounded down to the next smaller multiple of 8. ^(The lookaside memory +** configuration for a database connection can only be changed when that +** connection is not currently using lookaside memory, or in other words +** when the "current value" returned by +** [sqlite3_db_status](D,[SQLITE_CONFIG_LOOKASIDE],...) is zero. +** Any attempt to change the lookaside memory configuration when lookaside +** memory is in use leaves the configuration unchanged and returns +** [SQLITE_BUSY].)^</dd> ** ** </dl> */ #define SQLITE_DBCONFIG_LOOKASIDE 1001 /* void* int int */ @@ -2250,10 +2276,13 @@ */ SQLITE_API int sqlite3_busy_timeout(sqlite3*, int ms); /* ** CAPI3REF: Convenience Routines For Running Queries +** +** This is a legacy interface that is preserved for backwards compatibility. +** Use of this interface is not recommended. ** ** Definition: A <b>result table</b> is memory data structure created by the ** [sqlite3_get_table()] interface. A result table records the ** complete query results from one or more queries. ** @@ -2271,11 +2300,11 @@ ** ** A result table might consist of one or more memory allocations. ** It is not safe to pass a result table directly to [sqlite3_free()]. ** A result table should be deallocated using [sqlite3_free_table()]. ** -** As an example of the result table format, suppose a query result +** ^(As an example of the result table format, suppose a query result ** is as follows: ** ** <blockquote><pre> ** Name | Age ** ----------------------- @@ -2295,31 +2324,31 @@ ** azResult[3] = "43"; ** azResult[4] = "Bob"; ** azResult[5] = "28"; ** azResult[6] = "Cindy"; ** azResult[7] = "21"; -** </pre></blockquote> +** </pre></blockquote>)^ ** ** ^The sqlite3_get_table() function evaluates one or more ** semicolon-separated SQL statements in the zero-terminated UTF-8 ** string of its 2nd parameter and returns a result table to the ** pointer given in its 3rd parameter. ** ** After the application has finished with the result from sqlite3_get_table(), -** it should pass the result table pointer to sqlite3_free_table() in order to +** it must pass the result table pointer to sqlite3_free_table() in order to ** release the memory that was malloced. Because of the way the ** [sqlite3_malloc()] happens within sqlite3_get_table(), the calling ** function must not try to call [sqlite3_free()] directly. Only ** [sqlite3_free_table()] is able to release the memory properly and safely. ** -** ^(The sqlite3_get_table() interface is implemented as a wrapper around +** The sqlite3_get_table() interface is implemented as a wrapper around ** [sqlite3_exec()]. The sqlite3_get_table() routine does not have access ** to any internal data structures of SQLite. It uses only the public ** interface defined here. As a consequence, errors that occur in the ** wrapper layer outside of the internal [sqlite3_exec()] call are not ** reflected in subsequent calls to [sqlite3_errcode()] or -** [sqlite3_errmsg()].)^ +** [sqlite3_errmsg()]. */ SQLITE_API int sqlite3_get_table( sqlite3 *db, /* An open database */ const char *zSql, /* SQL to be evaluated */ char ***pazResult, /* Results of the query */ @@ -2467,11 +2496,13 @@ ** by sqlite3_realloc() and the prior allocation is freed. ** ^If sqlite3_realloc() returns NULL, then the prior allocation ** is not freed. ** ** ^The memory returned by sqlite3_malloc() and sqlite3_realloc() -** is always aligned to at least an 8 byte boundary. +** is always aligned to at least an 8 byte boundary, or to a +** 4 byte boundary if the [SQLITE_4_BYTE_ALIGNED_MALLOC] compile-time +** option is used. ** ** In SQLite version 3.5.0 and 3.5.1, it was possible to define ** the SQLITE_OMIT_MEMORY_ALLOCATION which would cause the built-in ** implementation of these routines to be omitted. That capability ** is no longer provided. Only built-in memory allocators can be used. @@ -2725,21 +2756,32 @@ void(*xProfile)(void*,const char*,sqlite3_uint64), void*); /* ** CAPI3REF: Query Progress Callbacks ** -** ^This routine configures a callback function - the -** progress callback - that is invoked periodically during long -** running calls to [sqlite3_exec()], [sqlite3_step()] and -** [sqlite3_get_table()]. An example use for this +** ^The sqlite3_progress_handler(D,N,X,P) interface causes the callback +** function X to be invoked periodically during long running calls to +** [sqlite3_exec()], [sqlite3_step()] and [sqlite3_get_table()] for +** database connection D. An example use for this ** interface is to keep a GUI updated during a large query. +** +** ^The parameter P is passed through as the only parameter to the +** callback function X. ^The parameter N is the number of +** [virtual machine instructions] that are evaluated between successive +** invocations of the callback X. +** +** ^Only a single progress handler may be defined at one time per +** [database connection]; setting a new progress handler cancels the +** old one. ^Setting parameter X to NULL disables the progress handler. +** ^The progress handler is also disabled by setting N to a value less +** than 1. ** ** ^If the progress callback returns non-zero, the operation is ** interrupted. This feature can be used to implement a ** "Cancel" button on a GUI progress dialog box. ** -** The progress handler must not do anything that will modify +** The progress handler callback must not do anything that will modify ** the database connection that invoked the progress handler. ** Note that [sqlite3_prepare_v2()] and [sqlite3_step()] both modify their ** database connections for the meaning of "modify" in this paragraph. ** */ @@ -2794,11 +2836,11 @@ ** </dl> ** ** If the 3rd parameter to sqlite3_open_v2() is not one of the ** combinations shown above or one of the combinations shown above combined ** with the [SQLITE_OPEN_NOMUTEX], [SQLITE_OPEN_FULLMUTEX], -** [SQLITE_OPEN_SHAREDCACHE] and/or [SQLITE_OPEN_SHAREDCACHE] flags, +** [SQLITE_OPEN_SHAREDCACHE] and/or [SQLITE_OPEN_PRIVATECACHE] flags, ** then the behavior is undefined. ** ** ^If the [SQLITE_OPEN_NOMUTEX] flag is set, then the database connection ** opens in the multi-thread [threading mode] as long as the single-thread ** mode has not been set at compile-time or start-time. ^If the @@ -2919,21 +2961,26 @@ ** ^(This interface allows the size of various constructs to be limited ** on a connection by connection basis. The first parameter is the ** [database connection] whose limit is to be set or queried. The ** second parameter is one of the [limit categories] that define a ** class of constructs to be size limited. The third parameter is the -** new limit for that construct. The function returns the old limit.)^ +** new limit for that construct.)^ ** ** ^If the new limit is a negative number, the limit is unchanged. -** ^(For the limit category of SQLITE_LIMIT_XYZ there is a +** ^(For each limit category SQLITE_LIMIT_<i>NAME</i> there is a ** [limits | hard upper bound] -** set by a compile-time C preprocessor macro named -** [limits | SQLITE_MAX_XYZ]. +** set at compile-time by a C preprocessor macro called +** [limits | SQLITE_MAX_<i>NAME</i>]. ** (The "_LIMIT_" in the name is changed to "_MAX_".))^ ** ^Attempts to increase a limit above its hard upper bound are ** silently truncated to the hard upper bound. ** +** ^Regardless of whether or not the limit was changed, the +** [sqlite3_limit()] interface returns the prior value of the limit. +** ^Hence, to find the current value of a limit without changing it, +** simply invoke this interface with the third parameter set to -1. +** ** Run-time limits are intended for use in applications that manage ** both their own internal database and also databases that are controlled ** by untrusted external sources. An example application might be a ** web browser that has its own databases for storing history and ** separate databases controlled by JavaScript applications downloaded @@ -2958,11 +3005,11 @@ ** The synopsis of the meanings of the various limits is shown below. ** Additional information is available at [limits | Limits in SQLite]. ** ** <dl> ** ^(<dt>SQLITE_LIMIT_LENGTH</dt> -** <dd>The maximum size of any string or BLOB or table row.<dd>)^ +** <dd>The maximum size of any string or BLOB or table row, in bytes.<dd>)^ ** ** ^(<dt>SQLITE_LIMIT_SQL_LENGTH</dt> ** <dd>The maximum length of an SQL statement, in bytes.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_COLUMN</dt> @@ -2976,11 +3023,13 @@ ** ^(<dt>SQLITE_LIMIT_COMPOUND_SELECT</dt> ** <dd>The maximum number of terms in a compound SELECT statement.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_VDBE_OP</dt> ** <dd>The maximum number of instructions in a virtual machine program -** used to implement an SQL statement.</dd>)^ +** used to implement an SQL statement. This limit is not currently +** enforced, though that might be added in some future release of +** SQLite.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_FUNCTION_ARG</dt> ** <dd>The maximum number of arguments on a function.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_ATTACHED</dt> @@ -2989,12 +3038,11 @@ ** ^(<dt>SQLITE_LIMIT_LIKE_PATTERN_LENGTH</dt> ** <dd>The maximum length of the pattern argument to the [LIKE] or ** [GLOB] operators.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_VARIABLE_NUMBER</dt> -** <dd>The maximum number of variables in an SQL statement that can -** be bound.</dd>)^ +** <dd>The maximum index number of any [parameter] in an SQL statement.)^ ** ** ^(<dt>SQLITE_LIMIT_TRIGGER_DEPTH</dt> ** <dd>The maximum depth of recursion for triggers.</dd>)^ ** </dl> */ @@ -3062,16 +3110,11 @@ ** ** <ol> ** <li> ** ^If the database schema changes, instead of returning [SQLITE_SCHEMA] as it ** always used to do, [sqlite3_step()] will automatically recompile the SQL -** statement and try to run it again. ^If the schema has changed in -** a way that makes the statement no longer valid, [sqlite3_step()] will still -** return [SQLITE_SCHEMA]. But unlike the legacy behavior, [SQLITE_SCHEMA] is -** now a fatal error. Calling [sqlite3_prepare_v2()] again will not make the -** error go away. Note: use [sqlite3_errmsg()] to find the text -** of the parsing error that results in an [SQLITE_SCHEMA] return. +** statement and try to run it again. ** </li> ** ** <li> ** ^When an error occurs, [sqlite3_step()] will return one of the detailed ** [error codes] or [extended error codes]. ^The legacy behavior was that @@ -3080,15 +3123,20 @@ ** in order to find the underlying cause of the problem. With the "v2" prepare ** interfaces, the underlying reason for the error is returned immediately. ** </li> ** ** <li> -** ^If the value of a [parameter | host parameter] in the WHERE clause might -** change the query plan for a statement, then the statement may be -** automatically recompiled (as if there had been a schema change) on the first -** [sqlite3_step()] call following any change to the -** [sqlite3_bind_text | bindings] of the [parameter]. +** ^If the specific value bound to [parameter | host parameter] in the +** WHERE clause might influence the choice of query plan for a statement, +** then the statement will be automatically recompiled, as if there had been +** a schema change, on the first [sqlite3_step()] call following any change +** to the [sqlite3_bind_text | bindings] of that [parameter]. +** ^The specific value of WHERE-clause [parameter] might influence the +** choice of query plan if the parameter is the left-hand side of a [LIKE] +** or [GLOB] operator or if the parameter is compared to an indexed column +** and the [SQLITE_ENABLE_STAT2] compile-time option is enabled. +** the ** </li> ** </ol> */ SQLITE_API int sqlite3_prepare( sqlite3 *db, /* Database handle */ @@ -3151,11 +3199,11 @@ ** or if SQLite is run in one of reduced mutex modes ** [SQLITE_CONFIG_SINGLETHREAD] or [SQLITE_CONFIG_MULTITHREAD] ** then there is no distinction between protected and unprotected ** sqlite3_value objects and they can be used interchangeably. However, ** for maximum code portability it is recommended that applications -** still make the distinction between between protected and unprotected +** still make the distinction between protected and unprotected ** sqlite3_value objects even when not strictly required. ** ** ^The sqlite3_value objects that are passed as parameters into the ** implementation of [application-defined SQL functions] are protected. ** ^The sqlite3_value object returned by @@ -3197,11 +3245,11 @@ ** <li> @VVV ** <li> $VVV ** </ul> ** ** In the templates above, NNN represents an integer literal, -** and VVV represents an alphanumeric identifer.)^ ^The values of these +** and VVV represents an alphanumeric identifier.)^ ^The values of these ** parameters (also called "host parameter names" or "SQL parameters") ** can be set using the sqlite3_bind_*() routines defined here. ** ** ^The first argument to the sqlite3_bind_*() routines is always ** a pointer to the [sqlite3_stmt] object returned from @@ -3346,10 +3394,12 @@ ** CAPI3REF: Number Of Columns In A Result Set ** ** ^Return the number of columns in the result set returned by the ** [prepared statement]. ^This routine returns 0 if pStmt is an SQL ** statement that does not return data (for example an [UPDATE]). +** +** See also: [sqlite3_data_count()] */ SQLITE_API int sqlite3_column_count(sqlite3_stmt *pStmt); /* ** CAPI3REF: Column Names In A Result Set @@ -3536,12 +3586,18 @@ SQLITE_API int sqlite3_step(sqlite3_stmt*); /* ** CAPI3REF: Number of columns in a result set ** -** ^The sqlite3_data_count(P) the number of columns in the -** of the result set of [prepared statement] P. +** ^The sqlite3_data_count(P) interface returns the number of columns in the +** current row of the result set of [prepared statement] P. +** ^If prepared statement P does not have results ready to return +** (via calls to the [sqlite3_column_int | sqlite3_column_*()] of +** interfaces) then sqlite3_data_count(P) returns 0. +** ^The sqlite3_data_count(P) routine also returns 0 if P is a NULL pointer. +** +** See also: [sqlite3_column_count()] */ SQLITE_API int sqlite3_data_count(sqlite3_stmt *pStmt); /* ** CAPI3REF: Fundamental Datatypes @@ -3617,22 +3673,30 @@ ** ^If the result is a UTF-16 string, then sqlite3_column_bytes() converts ** the string to UTF-8 and then returns the number of bytes. ** ^If the result is a numeric value then sqlite3_column_bytes() uses ** [sqlite3_snprintf()] to convert that value to a UTF-8 string and returns ** the number of bytes in that string. -** ^The value returned does not include the zero terminator at the end -** of the string. ^For clarity: the value returned is the number of +** ^If the result is NULL, then sqlite3_column_bytes() returns zero. +** +** ^If the result is a BLOB or UTF-16 string then the sqlite3_column_bytes16() +** routine returns the number of bytes in that BLOB or string. +** ^If the result is a UTF-8 string, then sqlite3_column_bytes16() converts +** the string to UTF-16 and then returns the number of bytes. +** ^If the result is a numeric value then sqlite3_column_bytes16() uses +** [sqlite3_snprintf()] to convert that value to a UTF-16 string and returns +** the number of bytes in that string. +** ^If the result is NULL, then sqlite3_column_bytes16() returns zero. +** +** ^The values returned by [sqlite3_column_bytes()] and +** [sqlite3_column_bytes16()] do not include the zero terminators at the end +** of the string. ^For clarity: the values returned by +** [sqlite3_column_bytes()] and [sqlite3_column_bytes16()] are the number of ** bytes in the string, not the number of characters. ** ** ^Strings returned by sqlite3_column_text() and sqlite3_column_text16(), ** even empty strings, are always zero terminated. ^The return -** value from sqlite3_column_blob() for a zero-length BLOB is an arbitrary -** pointer, possibly even a NULL pointer. -** -** ^The sqlite3_column_bytes16() routine is similar to sqlite3_column_bytes() -** but leaves the result in UTF-16 in native byte order instead of UTF-8. -** ^The zero terminator is not included in this count. +** value from sqlite3_column_blob() for a zero-length BLOB is a NULL pointer. ** ** ^The object returned by [sqlite3_column_value()] is an ** [unprotected sqlite3_value] object. An unprotected sqlite3_value object ** may only be used with [sqlite3_bind_value()] and [sqlite3_result_value()]. ** If the [unprotected sqlite3_value] object returned by @@ -3673,14 +3737,14 @@ ** and atof(). SQLite does not really use these functions. It has its ** own equivalent internal routines. The atoi() and atof() names are ** used in the table for brevity and because they are familiar to most ** C programmers. ** -** ^Note that when type conversions occur, pointers returned by prior +** Note that when type conversions occur, pointers returned by prior ** calls to sqlite3_column_blob(), sqlite3_column_text(), and/or ** sqlite3_column_text16() may be invalidated. -** ^(Type conversions and pointer invalidations might occur +** Type conversions and pointer invalidations might occur ** in the following cases: ** ** <ul> ** <li> The initial content is a BLOB and sqlite3_column_text() or ** sqlite3_column_text16() is called. A zero-terminator might @@ -3689,26 +3753,26 @@ ** sqlite3_column_text16() is called. The content must be converted ** to UTF-16.</li> ** <li> The initial content is UTF-16 text and sqlite3_column_bytes() or ** sqlite3_column_text() is called. The content must be converted ** to UTF-8.</li> -** </ul>)^ +** </ul> ** ** ^Conversions between UTF-16be and UTF-16le are always done in place and do ** not invalidate a prior pointer, though of course the content of the buffer -** that the prior pointer points to will have been modified. Other kinds +** that the prior pointer references will have been modified. Other kinds ** of conversion are done in place when it is possible, but sometimes they ** are not possible and in those cases prior pointers are invalidated. ** -** ^(The safest and easiest to remember policy is to invoke these routines +** The safest and easiest to remember policy is to invoke these routines ** in one of the following ways: ** ** <ul> ** <li>sqlite3_column_text() followed by sqlite3_column_bytes()</li> ** <li>sqlite3_column_blob() followed by sqlite3_column_bytes()</li> ** <li>sqlite3_column_text16() followed by sqlite3_column_bytes16()</li> -** </ul>)^ +** </ul> ** ** In other words, you should call sqlite3_column_text(), ** sqlite3_column_blob(), or sqlite3_column_text16() first to force the result ** into the desired format, then invoke sqlite3_column_bytes() or ** sqlite3_column_bytes16() to find the size of the result. Do not mix calls @@ -3742,21 +3806,30 @@ /* ** CAPI3REF: Destroy A Prepared Statement Object ** ** ^The sqlite3_finalize() function is called to delete a [prepared statement]. -** ^If the statement was executed successfully or not executed at all, then -** SQLITE_OK is returned. ^If execution of the statement failed then an -** [error code] or [extended error code] is returned. +** ^If the most recent evaluation of the statement encountered no errors or +** or if the statement is never been evaluated, then sqlite3_finalize() returns +** SQLITE_OK. ^If the most recent evaluation of statement S failed, then +** sqlite3_finalize(S) returns the appropriate [error code] or +** [extended error code]. ** -** ^This routine can be called at any point during the execution of the -** [prepared statement]. ^If the virtual machine has not -** completed execution when this routine is called, that is like -** encountering an error or an [sqlite3_interrupt | interrupt]. -** ^Incomplete updates may be rolled back and transactions canceled, -** depending on the circumstances, and the -** [error code] returned will be [SQLITE_ABORT]. +** ^The sqlite3_finalize(S) routine can be called at any point during +** the life cycle of [prepared statement] S: +** before statement S is ever evaluated, after +** one or more calls to [sqlite3_reset()], or after any call +** to [sqlite3_step()] regardless of whether or not the statement has +** completed execution. +** +** ^Invoking sqlite3_finalize() on a NULL pointer is a harmless no-op. +** +** The application must finalize every [prepared statement] in order to avoid +** resource leaks. It is a grievous error for the application to try to use +** a prepared statement after it has been finalized. Any use of a prepared +** statement after it has been finalized can result in undefined and +** undesirable behavior such as segfaults and heap corruption. */ SQLITE_API int sqlite3_finalize(sqlite3_stmt *pStmt); /* ** CAPI3REF: Reset A Prepared Statement Object @@ -3788,40 +3861,42 @@ ** CAPI3REF: Create Or Redefine SQL Functions ** KEYWORDS: {function creation routines} ** KEYWORDS: {application-defined SQL function} ** KEYWORDS: {application-defined SQL functions} ** -** ^These two functions (collectively known as "function creation routines") +** ^These functions (collectively known as "function creation routines") ** are used to add SQL functions or aggregates or to redefine the behavior -** of existing SQL functions or aggregates. The only difference between the -** two is that the second parameter, the name of the (scalar) function or -** aggregate, is encoded in UTF-8 for sqlite3_create_function() and UTF-16 -** for sqlite3_create_function16(). +** of existing SQL functions or aggregates. The only differences between +** these routines are the text encoding expected for +** the the second parameter (the name of the function being created) +** and the presence or absence of a destructor callback for +** the application data pointer. ** ** ^The first parameter is the [database connection] to which the SQL ** function is to be added. ^If an application uses more than one database ** connection then application-defined SQL functions must be added ** to each database connection separately. ** -** The second parameter is the name of the SQL function to be created or -** redefined. ^The length of the name is limited to 255 bytes, exclusive of -** the zero-terminator. Note that the name length limit is in bytes, not -** characters. ^Any attempt to create a function with a longer name -** will result in [SQLITE_ERROR] being returned. +** ^The second parameter is the name of the SQL function to be created or +** redefined. ^The length of the name is limited to 255 bytes in a UTF-8 +** representation, exclusive of the zero-terminator. ^Note that the name +** length limit is in UTF-8 bytes, not characters nor UTF-16 bytes. +** ^Any attempt to create a function with a longer name +** will result in [SQLITE_MISUSE] being returned. ** ** ^The third parameter (nArg) ** is the number of arguments that the SQL function or ** aggregate takes. ^If this parameter is -1, then the SQL function or ** aggregate may take any number of arguments between 0 and the limit ** set by [sqlite3_limit]([SQLITE_LIMIT_FUNCTION_ARG]). If the third ** parameter is less than -1 or greater than 127 then the behavior is ** undefined. ** -** The fourth parameter, eTextRep, specifies what +** ^The fourth parameter, eTextRep, specifies what ** [SQLITE_UTF8 | text encoding] this SQL function prefers for -** its parameters. Any SQL function implementation should be able to work -** work with UTF-8, UTF-16le, or UTF-16be. But some implementations may be +** its parameters. Every SQL function implementation must be able to work +** with UTF-8, UTF-16le, or UTF-16be. But some implementations may be ** more efficient with one encoding than another. ^An application may ** invoke sqlite3_create_function() or sqlite3_create_function16() multiple ** times with the same function but with different values of eTextRep. ** ^When multiple implementations of the same function are available, SQLite ** will pick the one that involves the least amount of data conversion. @@ -3829,17 +3904,25 @@ ** encoding is used, then the fourth argument should be [SQLITE_ANY]. ** ** ^(The fifth parameter is an arbitrary pointer. The implementation of the ** function can gain access to this pointer using [sqlite3_user_data()].)^ ** -** The seventh, eighth and ninth parameters, xFunc, xStep and xFinal, are +** ^The seventh, eighth and ninth parameters, xFunc, xStep and xFinal, are ** pointers to C-language functions that implement the SQL function or ** aggregate. ^A scalar SQL function requires an implementation of the xFunc -** callback only; NULL pointers should be passed as the xStep and xFinal +** callback only; NULL pointers must be passed as the xStep and xFinal ** parameters. ^An aggregate SQL function requires an implementation of xStep -** and xFinal and NULL should be passed for xFunc. ^To delete an existing -** SQL function or aggregate, pass NULL for all three function callbacks. +** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing +** SQL function or aggregate, pass NULL poiners for all three function +** callbacks. +** +** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL, +** then it is invoked when the function is deleted, either by being +** overloaded or when the database connection closes. +** ^When the destructure callback of the tenth parameter is invoked, it +** is passed a single argument which is a copy of the pointer which was +** the fifth parameter to sqlite3_create_function_v2(). ** ** ^It is permitted to register multiple implementations of the same ** functions with the same name but with either differing numbers of ** arguments or differing preferred text encodings. ^SQLite will use ** the implementation that most closely matches the way in which the @@ -3851,15 +3934,10 @@ ** ^A function where the encoding difference is between UTF16le and UTF16be ** is a closer match than a function where the encoding difference is ** between UTF8 and UTF16. ** ** ^Built-in functions may be overloaded by new application-defined functions. -** ^The first application-defined function with a given name overrides all -** built-in functions in the same [database connection] with the same name. -** ^Subsequent application-defined functions of the same name only override -** prior application-defined functions that are an exact match for the -** number of parameters and preferred encoding. ** ** ^An application-defined function is permitted to call other ** SQLite interfaces. However, such calls must not ** close the database connection nor finalize or reset the prepared ** statement in which the function is running. @@ -3881,10 +3959,21 @@ int eTextRep, void *pApp, void (*xFunc)(sqlite3_context*,int,sqlite3_value**), void (*xStep)(sqlite3_context*,int,sqlite3_value**), void (*xFinal)(sqlite3_context*) +); +SQLITE_API int sqlite3_create_function_v2( + sqlite3 *db, + const char *zFunctionName, + int nArg, + int eTextRep, + void *pApp, + void (*xFunc)(sqlite3_context*,int,sqlite3_value**), + void (*xStep)(sqlite3_context*,int,sqlite3_value**), + void (*xFinal)(sqlite3_context*), + void(*xDestroy)(void*) ); /* ** CAPI3REF: Text Encodings ** @@ -3976,11 +4065,11 @@ SQLITE_API int sqlite3_value_numeric_type(sqlite3_value*); /* ** CAPI3REF: Obtain Aggregate Function Context ** -** Implementions of aggregate SQL functions use this +** Implementations of aggregate SQL functions use this ** routine to allocate memory for storing their state. ** ** ^The first time the sqlite3_aggregate_context(C,N) routine is called ** for a particular aggregate function, SQLite ** allocates N of memory, zeroes out that memory, and returns a pointer @@ -4228,73 +4317,97 @@ SQLITE_API void sqlite3_result_zeroblob(sqlite3_context*, int n); /* ** CAPI3REF: Define New Collating Sequences ** -** These functions are used to add new collation sequences to the -** [database connection] specified as the first argument. +** ^These functions add, remove, or modify a [collation] associated +** with the [database connection] specified as the first argument. ** -** ^The name of the new collation sequence is specified as a UTF-8 string +** ^The name of the collation is a UTF-8 string ** for sqlite3_create_collation() and sqlite3_create_collation_v2() -** and a UTF-16 string for sqlite3_create_collation16(). ^In all cases -** the name is passed as the second function argument. -** -** ^The third argument may be one of the constants [SQLITE_UTF8], -** [SQLITE_UTF16LE], or [SQLITE_UTF16BE], indicating that the user-supplied -** routine expects to be passed pointers to strings encoded using UTF-8, -** UTF-16 little-endian, or UTF-16 big-endian, respectively. ^The -** third argument might also be [SQLITE_UTF16] to indicate that the routine -** expects pointers to be UTF-16 strings in the native byte order, or the -** argument can be [SQLITE_UTF16_ALIGNED] if the -** the routine expects pointers to 16-bit word aligned strings -** of UTF-16 in the native byte order. -** -** A pointer to the user supplied routine must be passed as the fifth -** argument. ^If it is NULL, this is the same as deleting the collation -** sequence (so that SQLite cannot call it anymore). -** ^Each time the application supplied function is invoked, it is passed -** as its first parameter a copy of the void* passed as the fourth argument -** to sqlite3_create_collation() or sqlite3_create_collation16(). -** -** ^The remaining arguments to the application-supplied routine are two strings, -** each represented by a (length, data) pair and encoded in the encoding -** that was passed as the third argument when the collation sequence was -** registered. The application defined collation routine should -** return negative, zero or positive if the first string is less than, -** equal to, or greater than the second string. i.e. (STRING1 - STRING2). +** and a UTF-16 string in native byte order for sqlite3_create_collation16(). +** ^Collation names that compare equal according to [sqlite3_strnicmp()] are +** considered to be the same name. +** +** ^(The third argument (eTextRep) must be one of the constants: +** <ul> +** <li> [SQLITE_UTF8], +** <li> [SQLITE_UTF16LE], +** <li> [SQLITE_UTF16BE], +** <li> [SQLITE_UTF16], or +** <li> [SQLITE_UTF16_ALIGNED]. +** </ul>)^ +** ^The eTextRep argument determines the encoding of strings passed +** to the collating function callback, xCallback. +** ^The [SQLITE_UTF16] and [SQLITE_UTF16_ALIGNED] values for eTextRep +** force strings to be UTF16 with native byte order. +** ^The [SQLITE_UTF16_ALIGNED] value for eTextRep forces strings to begin +** on an even byte address. +** +** ^The fourth argument, pArg, is a application data pointer that is passed +** through as the first argument to the collating function callback. +** +** ^The fifth argument, xCallback, is a pointer to the collating function. +** ^Multiple collating functions can be registered using the same name but +** with different eTextRep parameters and SQLite will use whichever +** function requires the least amount of data transformation. +** ^If the xCallback argument is NULL then the collating function is +** deleted. ^When all collating functions having the same name are deleted, +** that collation is no longer usable. +** +** ^The collating function callback is invoked with a copy of the pArg +** application data pointer and with two strings in the encoding specified +** by the eTextRep argument. The collating function must return an +** integer that is negative, zero, or positive +** if the first string is less than, equal to, or greater than the second, +** respectively. A collating function must alway return the same answer +** given the same inputs. If two or more collating functions are registered +** to the same collation name (using different eTextRep values) then all +** must give an equivalent answer when invoked with equivalent strings. +** The collating function must obey the following properties for all +** strings A, B, and C: +** +** <ol> +** <li> If A==B then B==A. +** <li> If A==B and B==C then A==C. +** <li> If A<B THEN B>A. +** <li> If A<B and B<C then A<C. +** </ol> +** +** If a collating function fails any of the above constraints and that +** collating function is registered and used, then the behavior of SQLite +** is undefined. ** ** ^The sqlite3_create_collation_v2() works like sqlite3_create_collation() -** except that it takes an extra argument which is a destructor for -** the collation. ^The destructor is called when the collation is -** destroyed and is passed a copy of the fourth parameter void* pointer -** of the sqlite3_create_collation_v2(). -** ^Collations are destroyed when they are overridden by later calls to the -** collation creation functions or when the [database connection] is closed -** using [sqlite3_close()]. +** with the addition that the xDestroy callback is invoked on pArg when +** the collating function is deleted. +** ^Collating functions are deleted when they are overridden by later +** calls to the collation creation functions or when the +** [database connection] is closed using [sqlite3_close()]. ** ** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()]. */ SQLITE_API int sqlite3_create_collation( sqlite3*, const char *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*) ); SQLITE_API int sqlite3_create_collation_v2( sqlite3*, const char *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*), void(*xDestroy)(void*) ); SQLITE_API int sqlite3_create_collation16( sqlite3*, const void *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*) ); /* ** CAPI3REF: Collation Needed Callbacks @@ -4379,20 +4492,23 @@ #endif /* ** CAPI3REF: Suspend Execution For A Short Time ** -** ^The sqlite3_sleep() function causes the current thread to suspend execution +** The sqlite3_sleep() function causes the current thread to suspend execution ** for at least a number of milliseconds specified in its parameter. ** -** ^If the operating system does not support sleep requests with +** If the operating system does not support sleep requests with ** millisecond time resolution, then the time will be rounded up to -** the nearest second. ^The number of milliseconds of sleep actually +** the nearest second. The number of milliseconds of sleep actually ** requested from the operating system is returned. ** ** ^SQLite implements this interface by calling the xSleep() -** method of the default [sqlite3_vfs] object. +** method of the default [sqlite3_vfs] object. If the xSleep() method +** of the default VFS is not implemented correctly, or not implemented at +** all, then the behavior of sqlite3_sleep() may deviate from the description +** in the previous paragraphs. */ SQLITE_API int sqlite3_sleep(int); /* ** CAPI3REF: Name Of The Folder Holding Temporary Files @@ -4610,44 +4726,77 @@ ** of heap memory by deallocating non-essential memory allocations ** held by the database library. Memory used to cache database ** pages to improve performance is an example of non-essential memory. ** ^sqlite3_release_memory() returns the number of bytes actually freed, ** which might be more or less than the amount requested. +** ^The sqlite3_release_memory() routine is a no-op returning zero +** if SQLite is not compiled with [SQLITE_ENABLE_MEMORY_MANAGEMENT]. */ SQLITE_API int sqlite3_release_memory(int); /* ** CAPI3REF: Impose A Limit On Heap Size ** -** ^The sqlite3_soft_heap_limit() interface places a "soft" limit -** on the amount of heap memory that may be allocated by SQLite. -** ^If an internal allocation is requested that would exceed the -** soft heap limit, [sqlite3_release_memory()] is invoked one or -** more times to free up some space before the allocation is performed. -** -** ^The limit is called "soft" because if [sqlite3_release_memory()] -** cannot free sufficient memory to prevent the limit from being exceeded, -** the memory is allocated anyway and the current operation proceeds. -** -** ^A negative or zero value for N means that there is no soft heap limit and -** [sqlite3_release_memory()] will only be called when memory is exhausted. -** ^The default value for the soft heap limit is zero. -** -** ^(SQLite makes a best effort to honor the soft heap limit. -** But if the soft heap limit cannot be honored, execution will -** continue without error or notification.)^ This is why the limit is -** called a "soft" limit. It is advisory only. -** -** Prior to SQLite version 3.5.0, this routine only constrained the memory -** allocated by a single thread - the same thread in which this routine -** runs. Beginning with SQLite version 3.5.0, the soft heap limit is -** applied to all threads. The value specified for the soft heap limit -** is an upper bound on the total memory allocation for all threads. In -** version 3.5.0 there is no mechanism for limiting the heap usage for -** individual threads. -*/ -SQLITE_API void sqlite3_soft_heap_limit(int); +** ^The sqlite3_soft_heap_limit64() interface sets and/or queries the +** soft limit on the amount of heap memory that may be allocated by SQLite. +** ^SQLite strives to keep heap memory utilization below the soft heap +** limit by reducing the number of pages held in the page cache +** as heap memory usages approaches the limit. +** ^The soft heap limit is "soft" because even though SQLite strives to stay +** below the limit, it will exceed the limit rather than generate +** an [SQLITE_NOMEM] error. In other words, the soft heap limit +** is advisory only. +** +** ^The return value from sqlite3_soft_heap_limit64() is the size of +** the soft heap limit prior to the call. ^If the argument N is negative +** then no change is made to the soft heap limit. Hence, the current +** size of the soft heap limit can be determined by invoking +** sqlite3_soft_heap_limit64() with a negative argument. +** +** ^If the argument N is zero then the soft heap limit is disabled. +** +** ^(The soft heap limit is not enforced in the current implementation +** if one or more of following conditions are true: +** +** <ul> +** <li> The soft heap limit is set to zero. +** <li> Memory accounting is disabled using a combination of the +** [sqlite3_config]([SQLITE_CONFIG_MEMSTATUS],...) start-time option and +** the [SQLITE_DEFAULT_MEMSTATUS] compile-time option. +** <li> An alternative page cache implementation is specifed using +** [sqlite3_config]([SQLITE_CONFIG_PCACHE],...). +** <li> The page cache allocates from its own memory pool supplied +** by [sqlite3_config]([SQLITE_CONFIG_PAGECACHE],...) rather than +** from the heap. +** </ul>)^ +** +** Beginning with SQLite version 3.7.3, the soft heap limit is enforced +** regardless of whether or not the [SQLITE_ENABLE_MEMORY_MANAGEMENT] +** compile-time option is invoked. With [SQLITE_ENABLE_MEMORY_MANAGEMENT], +** the soft heap limit is enforced on every memory allocation. Without +** [SQLITE_ENABLE_MEMORY_MANAGEMENT], the soft heap limit is only enforced +** when memory is allocated by the page cache. Testing suggests that because +** the page cache is the predominate memory user in SQLite, most +** applications will achieve adequate soft heap limit enforcement without +** the use of [SQLITE_ENABLE_MEMORY_MANAGEMENT]. +** +** The circumstances under which SQLite will enforce the soft heap limit may +** changes in future releases of SQLite. +*/ +SQLITE_API sqlite3_int64 sqlite3_soft_heap_limit64(sqlite3_int64 N); + +/* +** CAPI3REF: Deprecated Soft Heap Limit Interface +** DEPRECATED +** +** This is a deprecated version of the [sqlite3_soft_heap_limit64()] +** interface. This routine is provided for historical compatibility +** only. All new applications should use the +** [sqlite3_soft_heap_limit64()] interface rather than this one. +*/ +SQLITE_API SQLITE_DEPRECATED void sqlite3_soft_heap_limit(int N); + /* ** CAPI3REF: Extract Metadata About A Column Of A Table ** ** ^This routine returns metadata about a specific column of a specific @@ -4767,38 +4916,51 @@ ** it back off again. */ SQLITE_API int sqlite3_enable_load_extension(sqlite3 *db, int onoff); /* -** CAPI3REF: Automatically Load An Extensions -** -** ^This API can be invoked at program startup in order to register -** one or more statically linked extensions that will be available -** to all new [database connections]. -** -** ^(This routine stores a pointer to the extension entry point -** in an array that is obtained from [sqlite3_malloc()]. That memory -** is deallocated by [sqlite3_reset_auto_extension()].)^ -** -** ^This function registers an extension entry point that is -** automatically invoked whenever a new [database connection] -** is opened using [sqlite3_open()], [sqlite3_open16()], -** or [sqlite3_open_v2()]. -** ^Duplicate extensions are detected so calling this routine -** multiple times with the same extension is harmless. -** ^Automatic extensions apply across all threads. +** CAPI3REF: Automatically Load Statically Linked Extensions +** +** ^This interface causes the xEntryPoint() function to be invoked for +** each new [database connection] that is created. The idea here is that +** xEntryPoint() is the entry point for a statically linked SQLite extension +** that is to be automatically loaded into all new database connections. +** +** ^(Even though the function prototype shows that xEntryPoint() takes +** no arguments and returns void, SQLite invokes xEntryPoint() with three +** arguments and expects and integer result as if the signature of the +** entry point where as follows: +** +** <blockquote><pre> +** int xEntryPoint( +** sqlite3 *db, +** const char **pzErrMsg, +** const struct sqlite3_api_routines *pThunk +** ); +** </pre></blockquote>)^ +** +** If the xEntryPoint routine encounters an error, it should make *pzErrMsg +** point to an appropriate error message (obtained from [sqlite3_mprintf()]) +** and return an appropriate [error code]. ^SQLite ensures that *pzErrMsg +** is NULL before calling the xEntryPoint(). ^SQLite will invoke +** [sqlite3_free()] on *pzErrMsg after xEntryPoint() returns. ^If any +** xEntryPoint() returns an error, the [sqlite3_open()], [sqlite3_open16()], +** or [sqlite3_open_v2()] call that provoked the xEntryPoint() will fail. +** +** ^Calling sqlite3_auto_extension(X) with an entry point X that is already +** on the list of automatic extensions is a harmless no-op. ^No entry point +** will be called more than once for each database connection that is opened. +** +** See also: [sqlite3_reset_auto_extension()]. */ SQLITE_API int sqlite3_auto_extension(void (*xEntryPoint)(void)); /* ** CAPI3REF: Reset Automatic Extension Loading ** -** ^(This function disables all previously registered automatic -** extensions. It undoes the effect of all prior -** [sqlite3_auto_extension()] calls.)^ -** -** ^This function disables automatic extensions in all threads. +** ^This interface disables all automatic extensions previously +** registered using [sqlite3_auto_extension()]. */ SQLITE_API void sqlite3_reset_auto_extension(void); /* ** The interface to the virtual-table mechanism is currently considered @@ -5433,11 +5595,11 @@ ** output variable when querying the system for the current mutex ** implementation, using the [SQLITE_CONFIG_GETMUTEX] option. ** ** ^The xMutexInit method defined by this structure is invoked as ** part of system initialization by the sqlite3_initialize() function. -** ^The xMutexInit routine is calle by SQLite exactly once for each +** ^The xMutexInit routine is called by SQLite exactly once for each ** effective call to [sqlite3_initialize()]. ** ** ^The xMutexEnd method defined by this structure is invoked as ** part of system shutdown by the sqlite3_shutdown() function. The ** implementation of this method is expected to release all outstanding @@ -5466,11 +5628,11 @@ ** of passing a NULL pointer instead of a valid mutex handle are undefined ** (i.e. it is acceptable to provide an implementation that segfaults if ** it is passed a NULL pointer). ** ** The xMutexInit() method must be threadsafe. ^It must be harmless to -** invoke xMutexInit() mutiple times within the same process and without +** invoke xMutexInit() multiple times within the same process and without ** intervening calls to xMutexEnd(). Second and subsequent calls to ** xMutexInit() must be no-ops. ** ** ^xMutexInit() must not use SQLite memory allocation ([sqlite3_malloc()] ** and its associates). ^Similarly, xMutexAlloc() must not use SQLite memory @@ -5630,17 +5792,18 @@ #define SQLITE_TESTCTRL_ALWAYS 13 #define SQLITE_TESTCTRL_RESERVE 14 #define SQLITE_TESTCTRL_OPTIMIZATIONS 15 #define SQLITE_TESTCTRL_ISKEYWORD 16 #define SQLITE_TESTCTRL_PGHDRSZ 17 -#define SQLITE_TESTCTRL_LAST 17 +#define SQLITE_TESTCTRL_SCRATCHMALLOC 18 +#define SQLITE_TESTCTRL_LAST 18 /* ** CAPI3REF: SQLite Runtime Status ** ** ^This interface is used to retrieve runtime status information -** about the preformance of SQLite, and optionally to reset various +** about the performance of SQLite, and optionally to reset various ** highwater marks. ^The first argument is an integer code for ** the specific parameter to measure. ^(Recognized integer codes ** are of the form [SQLITE_STATUS_MEMORY_USED | SQLITE_STATUS_...].)^ ** ^The current value of the parameter is returned into *pCurrent. ** ^The highest recorded value is returned in *pHighwater. ^If the @@ -5649,11 +5812,11 @@ ** value. For those parameters ** nothing is written into *pHighwater and the resetFlag is ignored.)^ ** ^(Other parameters record only the highwater mark and not the current ** value. For these latter parameters nothing is written into *pCurrent.)^ ** -** ^The sqlite3_db_status() routine returns SQLITE_OK on success and a +** ^The sqlite3_status() routine returns SQLITE_OK on success and a ** non-zero [error code] on failure. ** ** This routine is threadsafe but is not atomic. This routine can be ** called while other threads are running the same or different SQLite ** interfaces. However the values returned in *pCurrent and @@ -5699,11 +5862,11 @@ ** [SQLITE_CONFIG_PAGECACHE]. The ** value returned is in pages, not in bytes.</dd>)^ ** ** ^(<dt>SQLITE_STATUS_PAGECACHE_OVERFLOW</dt> ** <dd>This parameter returns the number of bytes of page cache -** allocation which could not be statisfied by the [SQLITE_CONFIG_PAGECACHE] +** allocation which could not be satisfied by the [SQLITE_CONFIG_PAGECACHE] ** buffer and where forced to overflow to [sqlite3_malloc()]. The ** returned value includes allocations that overflowed because they ** where too large (they were larger than the "sz" parameter to ** [SQLITE_CONFIG_PAGECACHE]) and allocations that overflowed because ** no space was left in the page cache.</dd>)^ @@ -5722,11 +5885,11 @@ ** outstanding at time, this parameter also reports the number of threads ** using scratch memory at the same time.</dd>)^ ** ** ^(<dt>SQLITE_STATUS_SCRATCH_OVERFLOW</dt> ** <dd>This parameter returns the number of bytes of scratch memory -** allocation which could not be statisfied by the [SQLITE_CONFIG_SCRATCH] +** allocation which could not be satisfied by the [SQLITE_CONFIG_SCRATCH] ** buffer and where forced to overflow to [sqlite3_malloc()]. The values ** returned include overflows because the requested allocation was too ** larger (that is, because the requested allocation was larger than the ** "sz" parameter to [SQLITE_CONFIG_SCRATCH]) and because no scratch buffer ** slots were available. @@ -5762,18 +5925,21 @@ ** ^This interface is used to retrieve runtime status information ** about a single [database connection]. ^The first argument is the ** database connection object to be interrogated. ^The second argument ** is an integer constant, taken from the set of ** [SQLITE_DBSTATUS_LOOKASIDE_USED | SQLITE_DBSTATUS_*] macros, that -** determiness the parameter to interrogate. The set of +** determines the parameter to interrogate. The set of ** [SQLITE_DBSTATUS_LOOKASIDE_USED | SQLITE_DBSTATUS_*] macros is likely ** to grow in future releases of SQLite. ** ** ^The current value of the requested parameter is written into *pCur ** and the highest instantaneous value is written into *pHiwtr. ^If ** the resetFlg is true, then the highest instantaneous value is ** reset back down to the current value. +** +** ^The sqlite3_db_status() routine returns SQLITE_OK on success and a +** non-zero [error code] on failure. ** ** See also: [sqlite3_status()] and [sqlite3_stmt_status()]. */ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int resetFlg); @@ -5897,122 +6063,134 @@ ** CAPI3REF: Application Defined Page Cache. ** KEYWORDS: {page cache} ** ** ^(The [sqlite3_config]([SQLITE_CONFIG_PCACHE], ...) interface can ** register an alternative page cache implementation by passing in an -** instance of the sqlite3_pcache_methods structure.)^ The majority of the -** heap memory used by SQLite is used by the page cache to cache data read -** from, or ready to be written to, the database file. By implementing a -** custom page cache using this API, an application can control more -** precisely the amount of memory consumed by SQLite, the way in which +** instance of the sqlite3_pcache_methods structure.)^ +** In many applications, most of the heap memory allocated by +** SQLite is used for the page cache. +** By implementing a +** custom page cache using this API, an application can better control +** the amount of memory consumed by SQLite, the way in which ** that memory is allocated and released, and the policies used to ** determine exactly which parts of a database file are cached and for ** how long. ** +** The alternative page cache mechanism is an +** extreme measure that is only needed by the most demanding applications. +** The built-in page cache is recommended for most uses. +** ** ^(The contents of the sqlite3_pcache_methods structure are copied to an ** internal buffer by SQLite within the call to [sqlite3_config]. Hence ** the application may discard the parameter after the call to ** [sqlite3_config()] returns.)^ ** -** ^The xInit() method is called once for each call to [sqlite3_initialize()] +** ^(The xInit() method is called once for each effective +** call to [sqlite3_initialize()])^ ** (usually only once during the lifetime of the process). ^(The xInit() ** method is passed a copy of the sqlite3_pcache_methods.pArg value.)^ -** ^The xInit() method can set up up global structures and/or any mutexes +** The intent of the xInit() method is to set up global data structures ** required by the custom page cache implementation. +** ^(If the xInit() method is NULL, then the +** built-in default page cache is used instead of the application defined +** page cache.)^ ** -** ^The xShutdown() method is called from within [sqlite3_shutdown()], -** if the application invokes this API. It can be used to clean up +** ^The xShutdown() method is called by [sqlite3_shutdown()]. +** It can be used to clean up ** any outstanding resources before process shutdown, if required. +** ^The xShutdown() method may be NULL. ** -** ^SQLite holds a [SQLITE_MUTEX_RECURSIVE] mutex when it invokes -** the xInit method, so the xInit method need not be threadsafe. ^The +** ^SQLite automatically serializes calls to the xInit method, +** so the xInit method need not be threadsafe. ^The ** xShutdown method is only called from [sqlite3_shutdown()] so it does ** not need to be threadsafe either. All other methods must be threadsafe ** in multithreaded applications. ** ** ^SQLite will never invoke xInit() more than once without an intervening ** call to xShutdown(). ** -** ^The xCreate() method is used to construct a new cache instance. SQLite -** will typically create one cache instance for each open database file, +** ^SQLite invokes the xCreate() method to construct a new cache instance. +** SQLite will typically create one cache instance for each open database file, ** though this is not guaranteed. ^The ** first parameter, szPage, is the size in bytes of the pages that must ** be allocated by the cache. ^szPage will not be a power of two. ^szPage ** will the page size of the database file that is to be cached plus an -** increment (here called "R") of about 100 or 200. ^SQLite will use the +** increment (here called "R") of about 100 or 200. SQLite will use the ** extra R bytes on each page to store metadata about the underlying ** database page on disk. The value of R depends ** on the SQLite version, the target platform, and how SQLite was compiled. ** ^R is constant for a particular build of SQLite. ^The second argument to ** xCreate(), bPurgeable, is true if the cache being created will ** be used to cache database pages of a file stored on disk, or -** false if it is used for an in-memory database. ^The cache implementation +** false if it is used for an in-memory database. The cache implementation ** does not have to do anything special based with the value of bPurgeable; ** it is purely advisory. ^On a cache where bPurgeable is false, SQLite will ** never invoke xUnpin() except to deliberately delete a page. -** ^In other words, a cache created with bPurgeable set to false will +** ^In other words, calls to xUnpin() on a cache with bPurgeable set to +** false will always have the "discard" flag set to true. +** ^Hence, a cache created with bPurgeable false will ** never contain any unpinned pages. ** ** ^(The xCachesize() method may be called at any time by SQLite to set the ** suggested maximum cache-size (number of pages stored by) the cache ** instance passed as the first argument. This is the value configured using -** the SQLite "[PRAGMA cache_size]" command.)^ ^As with the bPurgeable +** the SQLite "[PRAGMA cache_size]" command.)^ As with the bPurgeable ** parameter, the implementation is not required to do anything with this ** value; it is advisory only. ** -** ^The xPagecount() method should return the number of pages currently -** stored in the cache. +** The xPagecount() method must return the number of pages currently +** stored in the cache, both pinned and unpinned. ** -** ^The xFetch() method is used to fetch a page and return a pointer to it. -** ^A 'page', in this context, is a buffer of szPage bytes aligned at an -** 8-byte boundary. ^The page to be fetched is determined by the key. ^The -** mimimum key value is 1. After it has been retrieved using xFetch, the page +** The xFetch() method locates a page in the cache and returns a pointer to +** the page, or a NULL pointer. +** A "page", in this context, means a buffer of szPage bytes aligned at an +** 8-byte boundary. The page to be fetched is determined by the key. ^The +** mimimum key value is 1. After it has been retrieved using xFetch, the page ** is considered to be "pinned". ** -** ^If the requested page is already in the page cache, then the page cache +** If the requested page is already in the page cache, then the page cache ** implementation must return a pointer to the page buffer with its content -** intact. ^(If the requested page is not already in the cache, then the -** behavior of the cache implementation is determined by the value of the -** createFlag parameter passed to xFetch, according to the following table: +** intact. If the requested page is not already in the cache, then the +** behavior of the cache implementation should use the value of the createFlag +** parameter to help it determined what action to take: ** ** <table border=1 width=85% align=center> ** <tr><th> createFlag <th> Behaviour when page is not already in cache ** <tr><td> 0 <td> Do not allocate a new page. Return NULL. ** <tr><td> 1 <td> Allocate a new page if it easy and convenient to do so. ** Otherwise return NULL. ** <tr><td> 2 <td> Make every effort to allocate a new page. Only return ** NULL if allocating a new page is effectively impossible. -** </table>)^ +** </table> ** -** SQLite will normally invoke xFetch() with a createFlag of 0 or 1. If -** a call to xFetch() with createFlag==1 returns NULL, then SQLite will +** ^(SQLite will normally invoke xFetch() with a createFlag of 0 or 1. SQLite +** will only use a createFlag of 2 after a prior call with a createFlag of 1 +** failed.)^ In between the to xFetch() calls, SQLite may ** attempt to unpin one or more cache pages by spilling the content of -** pinned pages to disk and synching the operating system disk cache. After -** attempting to unpin pages, the xFetch() method will be invoked again with -** a createFlag of 2. +** pinned pages to disk and synching the operating system disk cache. ** ** ^xUnpin() is called by SQLite with a pointer to a currently pinned page -** as its second argument. ^(If the third parameter, discard, is non-zero, -** then the page should be evicted from the cache. In this case SQLite -** assumes that the next time the page is retrieved from the cache using -** the xFetch() method, it will be zeroed.)^ ^If the discard parameter is -** zero, then the page is considered to be unpinned. ^The cache implementation +** as its second argument. If the third parameter, discard, is non-zero, +** then the page must be evicted from the cache. +** ^If the discard parameter is +** zero, then the page may be discarded or retained at the discretion of +** page cache implementation. ^The page cache implementation ** may choose to evict unpinned pages at any time. ** -** ^(The cache is not required to perform any reference counting. A single +** The cache must not perform any reference counting. A single ** call to xUnpin() unpins the page regardless of the number of prior calls -** to xFetch().)^ +** to xFetch(). ** -** ^The xRekey() method is used to change the key value associated with the -** page passed as the second argument from oldKey to newKey. ^If the cache -** previously contains an entry associated with newKey, it should be +** The xRekey() method is used to change the key value associated with the +** page passed as the second argument. If the cache +** previously contains an entry associated with newKey, it must be ** discarded. ^Any prior cache entry associated with newKey is guaranteed not ** to be pinned. ** -** ^When SQLite calls the xTruncate() method, the cache must discard all +** When SQLite calls the xTruncate() method, the cache must discard all ** existing cache entries with page numbers (keys) greater than or equal -** to the value of the iLimit parameter passed to xTruncate(). ^If any +** to the value of the iLimit parameter passed to xTruncate(). If any ** of these pages are pinned, they are implicitly unpinned, meaning that ** they can be safely discarded. ** ** ^The xDestroy() method is used to delete a cache allocated by xCreate(). ** All resources associated with the specified cache should be freed. ^After @@ -6184,11 +6362,11 @@ ** ** <b>sqlite3_backup_remaining(), sqlite3_backup_pagecount()</b> ** ** ^Each call to sqlite3_backup_step() sets two values inside ** the [sqlite3_backup] object: the number of pages still to be backed -** up and the total number of pages in the source databae file. +** up and the total number of pages in the source database file. ** The sqlite3_backup_remaining() and sqlite3_backup_pagecount() interfaces ** retrieve these two values, respectively. ** ** ^The values returned by these functions are only updated by ** sqlite3_backup_step(). ^If the source database is modified during a backup @@ -6280,11 +6458,11 @@ ** ^(There may be at most one unlock-notify callback registered by a ** blocked connection. If sqlite3_unlock_notify() is called when the ** blocked connection already has a registered unlock-notify callback, ** then the new callback replaces the old.)^ ^If sqlite3_unlock_notify() is ** called with a NULL pointer as its second argument, then any existing -** unlock-notify callback is cancelled. ^The blocked connections +** unlock-notify callback is canceled. ^The blocked connections ** unlock-notify callback may also be canceled by closing the blocked ** connection using [sqlite3_close()]. ** ** The unlock-notify callback is not reentrant. If an application invokes ** any sqlite3_xxx API functions from within an unlock-notify callback, a @@ -6362,11 +6540,11 @@ /* ** CAPI3REF: String Comparison ** ** ^The [sqlite3_strnicmp()] API allows applications and extensions to ** compare the contents of two buffers containing UTF-8 strings in a -** case-indendent fashion, using the same definition of case independence +** case-independent fashion, using the same definition of case independence ** that SQLite uses internally when comparing identifiers. */ SQLITE_API int sqlite3_strnicmp(const char *, const char *, int); /* @@ -6486,10 +6664,66 @@ #if 0 } /* End of the 'extern "C"' block */ #endif #endif +/* +** 2010 August 30 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +*/ + +#ifndef _SQLITE3RTREE_H_ +#define _SQLITE3RTREE_H_ + + +#if 0 +extern "C" { +#endif + +typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry; + +/* +** Register a geometry callback named zGeom that can be used as part of an +** R-Tree geometry query as follows: +** +** SELECT ... FROM <rtree> WHERE <rtree col> MATCH $zGeom(... params ...) +*/ +SQLITE_API int sqlite3_rtree_geometry_callback( + sqlite3 *db, + const char *zGeom, + int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes), + void *pContext +); + + +/* +** A pointer to a structure of the following type is passed as the first +** argument to callbacks registered using rtree_geometry_callback(). +*/ +struct sqlite3_rtree_geometry { + void *pContext; /* Copy of pContext passed to s_r_g_c() */ + int nParam; /* Size of array aParam[] */ + double *aParam; /* Parameters passed to SQL geom function */ + void *pUser; /* Callback implementation user data */ + void (*xDelUser)(void *); /* Called by SQLite to clean up pUser */ +}; + + +#if 0 +} /* end of the 'extern "C"' block */ +#endif + +#endif /* ifndef _SQLITE3RTREE_H_ */ + /************** End of sqlite3.h *********************************************/ /************** Continuing where we left off in sqliteInt.h ******************/ /************** Include hash.h in the middle of sqliteInt.h ******************/ /************** Begin file hash.h ********************************************/ @@ -7056,10 +7290,11 @@ typedef struct Schema Schema; typedef struct Expr Expr; typedef struct ExprList ExprList; typedef struct ExprSpan ExprSpan; typedef struct FKey FKey; +typedef struct FuncDestructor FuncDestructor; typedef struct FuncDef FuncDef; typedef struct FuncDefHash FuncDefHash; typedef struct IdList IdList; typedef struct Index Index; typedef struct IndexSample IndexSample; @@ -7162,16 +7397,15 @@ ** following values. ** ** NOTE: These values must match the corresponding PAGER_ values in ** pager.h. */ -#define BTREE_OMIT_JOURNAL 1 /* Do not use journal. No argument */ +#define BTREE_OMIT_JOURNAL 1 /* Do not create or use a rollback journal */ #define BTREE_NO_READLOCK 2 /* Omit readlocks on readonly files */ -#define BTREE_MEMORY 4 /* In-memory DB. No argument */ -#define BTREE_READONLY 8 /* Open the database in read-only mode */ -#define BTREE_READWRITE 16 /* Open for both reading and writing */ -#define BTREE_CREATE 32 /* Create the database if it does not exist */ +#define BTREE_MEMORY 4 /* This is an in-memory DB */ +#define BTREE_SINGLE 8 /* The file contains at most 1 b-tree */ +#define BTREE_UNORDERED 16 /* Use of a hash implementation is OK */ SQLITE_PRIVATE int sqlite3BtreeClose(Btree*); SQLITE_PRIVATE int sqlite3BtreeSetCacheSize(Btree*,int); SQLITE_PRIVATE int sqlite3BtreeSetSafetyLevel(Btree*,int,int); SQLITE_PRIVATE int sqlite3BtreeSyncDisabled(Btree*); @@ -7203,15 +7437,21 @@ SQLITE_PRIVATE int sqlite3BtreeCopyFile(Btree *, Btree *); SQLITE_PRIVATE int sqlite3BtreeIncrVacuum(Btree *); /* The flags parameter to sqlite3BtreeCreateTable can be the bitwise OR -** of the following flags: +** of the flags shown below. +** +** Every SQLite table must have either BTREE_INTKEY or BTREE_BLOBKEY set. +** With BTREE_INTKEY, the table key is a 64-bit integer and arbitrary data +** is stored in the leaves. (BTREE_INTKEY is used for SQL tables.) With +** BTREE_BLOBKEY, the key is an arbitrary BLOB and no content is stored +** anywhere - the key is the content. (BTREE_BLOBKEY is used for SQL +** indices.) */ #define BTREE_INTKEY 1 /* Table has only 64-bit signed integer keys */ -#define BTREE_ZERODATA 2 /* Table has keys only - no data */ -#define BTREE_LEAFDATA 4 /* Data stored in leaves only. Implies INTKEY */ +#define BTREE_BLOBKEY 2 /* Table has keys only - no data */ SQLITE_PRIVATE int sqlite3BtreeDropTable(Btree*, int, int*); SQLITE_PRIVATE int sqlite3BtreeClearTable(Btree*, int, int*); SQLITE_PRIVATE void sqlite3BtreeTripAllCursors(Btree*, int); @@ -7828,10 +8068,11 @@ ** ** NOTE: These values must match the corresponding BTREE_ values in btree.h. */ #define PAGER_OMIT_JOURNAL 0x0001 /* Do not use a rollback journal */ #define PAGER_NO_READLOCK 0x0002 /* Omit readlocks on readonly files */ +#define PAGER_MEMORY 0x0004 /* In-memory database */ /* ** Valid values for the second argument to sqlite3PagerLockingMode(). */ #define PAGER_LOCKINGMODE_QUERY -1 @@ -7868,11 +8109,11 @@ SQLITE_PRIVATE int sqlite3PagerClose(Pager *pPager); SQLITE_PRIVATE int sqlite3PagerReadFileheader(Pager*, int, unsigned char*); /* Functions used to configure a Pager object. */ SQLITE_PRIVATE void sqlite3PagerSetBusyhandler(Pager*, int(*)(void *), void *); -SQLITE_PRIVATE int sqlite3PagerSetPagesize(Pager*, u16*, int); +SQLITE_PRIVATE int sqlite3PagerSetPagesize(Pager*, u32*, int); SQLITE_PRIVATE int sqlite3PagerMaxPageCount(Pager*, int); SQLITE_PRIVATE void sqlite3PagerSetCachesize(Pager*, int); SQLITE_PRIVATE void sqlite3PagerSetSafetyLevel(Pager*,int,int); SQLITE_PRIVATE int sqlite3PagerLockingMode(Pager *, int); SQLITE_PRIVATE int sqlite3PagerSetJournalMode(Pager *, int); @@ -7895,11 +8136,11 @@ SQLITE_PRIVATE int sqlite3PagerPageRefcount(DbPage*); SQLITE_PRIVATE void *sqlite3PagerGetData(DbPage *); SQLITE_PRIVATE void *sqlite3PagerGetExtra(DbPage *); /* Functions used to manage pager transactions and savepoints. */ -SQLITE_PRIVATE int sqlite3PagerPagecount(Pager*, int*); +SQLITE_PRIVATE void sqlite3PagerPagecount(Pager*, int*); SQLITE_PRIVATE int sqlite3PagerBegin(Pager*, int exFlag, int); SQLITE_PRIVATE int sqlite3PagerCommitPhaseOne(Pager*,const char *zMaster, int); SQLITE_PRIVATE int sqlite3PagerExclusiveLock(Pager*); SQLITE_PRIVATE int sqlite3PagerSync(Pager *pPager); SQLITE_PRIVATE int sqlite3PagerCommitPhaseTwo(Pager*); @@ -8462,12 +8703,12 @@ #define sqlite3_mutex_alloc(X) ((sqlite3_mutex*)8) #define sqlite3_mutex_free(X) #define sqlite3_mutex_enter(X) #define sqlite3_mutex_try(X) SQLITE_OK #define sqlite3_mutex_leave(X) -#define sqlite3_mutex_held(X) 1 -#define sqlite3_mutex_notheld(X) 1 +#define sqlite3_mutex_held(X) ((void)(X),1) +#define sqlite3_mutex_notheld(X) ((void)(X),1) #define sqlite3MutexAlloc(X) ((sqlite3_mutex*)8) #define sqlite3MutexInit() SQLITE_OK #define sqlite3MutexEnd() #endif /* defined(SQLITE_MUTEX_OMIT) */ @@ -8785,10 +9026,31 @@ void (*xFunc)(sqlite3_context*,int,sqlite3_value**); /* Regular function */ void (*xStep)(sqlite3_context*,int,sqlite3_value**); /* Aggregate step */ void (*xFinalize)(sqlite3_context*); /* Aggregate finalizer */ char *zName; /* SQL name of the function. */ FuncDef *pHash; /* Next with a different name but the same hash */ + FuncDestructor *pDestructor; /* Reference counted destructor function */ +}; + +/* +** This structure encapsulates a user-function destructor callback (as +** configured using create_function_v2()) and a reference counter. When +** create_function_v2() is called to create a function with a destructor, +** a single object of this type is allocated. FuncDestructor.nRef is set to +** the number of FuncDef objects created (either 1 or 3, depending on whether +** or not the specified encoding is SQLITE_ANY). The FuncDef.pDestructor +** member of each of the new FuncDef objects is set to point to the allocated +** FuncDestructor. +** +** Thereafter, when one of the FuncDef objects is deleted, the reference +** count on this object is decremented. When it reaches 0, the destructor +** is invoked and the FuncDestructor structure freed. +*/ +struct FuncDestructor { + int nRef; + void (*xDestroy)(void *); + void *pUserData; }; /* ** Possible values for FuncDef.flags */ @@ -8825,19 +9087,19 @@ ** FuncDef.flags variable is set to the value passed as the flags ** parameter. */ #define FUNCTION(zName, nArg, iArg, bNC, xFunc) \ {nArg, SQLITE_UTF8, bNC*SQLITE_FUNC_NEEDCOLL, \ - SQLITE_INT_TO_PTR(iArg), 0, xFunc, 0, 0, #zName, 0} + SQLITE_INT_TO_PTR(iArg), 0, xFunc, 0, 0, #zName, 0, 0} #define STR_FUNCTION(zName, nArg, pArg, bNC, xFunc) \ {nArg, SQLITE_UTF8, bNC*SQLITE_FUNC_NEEDCOLL, \ - pArg, 0, xFunc, 0, 0, #zName, 0} + pArg, 0, xFunc, 0, 0, #zName, 0, 0} #define LIKEFUNC(zName, nArg, arg, flags) \ - {nArg, SQLITE_UTF8, flags, (void *)arg, 0, likeFunc, 0, 0, #zName, 0} + {nArg, SQLITE_UTF8, flags, (void *)arg, 0, likeFunc, 0, 0, #zName, 0, 0} #define AGGREGATE(zName, nArg, arg, nc, xStep, xFinal) \ {nArg, SQLITE_UTF8, nc*SQLITE_FUNC_NEEDCOLL, \ - SQLITE_INT_TO_PTR(arg), 0, 0, xStep,xFinal,#zName,0} + SQLITE_INT_TO_PTR(arg), 0, 0, xStep,xFinal,#zName,0,0} /* ** All current savepoints are stored in a linked list starting at ** sqlite3.pSavepoint. The first element in the list is the most recently ** opened savepoint. Savepoints are added to the list by the vdbe @@ -9053,10 +9315,11 @@ int iPKey; /* If not negative, use aCol[iPKey] as the primary key */ int nCol; /* Number of columns in this table */ Column *aCol; /* Information about each column */ Index *pIndex; /* List of SQL indexes on this table. */ int tnum; /* Root BTree node for this table (see note above) */ + unsigned nRowEst; /* Estimated rows in table - from sqlite_stat1 table */ Select *pSelect; /* NULL for tables. Points to definition if a view. */ u16 nRef; /* Number of pointers to this Table */ u8 tabFlags; /* Mask of TF_* values */ u8 keyConf; /* What to do in case of uniqueness conflict on iPKey */ FKey *pFKey; /* Linked list of all foreign keys in this table */ @@ -9183,13 +9446,13 @@ ** argument to sqlite3VdbeKeyCompare and is used to control the ** comparison of the two index keys. */ struct KeyInfo { sqlite3 *db; /* The database connection */ - u8 enc; /* Text encoding - one of the TEXT_Utf* values */ + u8 enc; /* Text encoding - one of the SQLITE_UTF* values */ u16 nField; /* Number of entries in aColl[] */ - u8 *aSortOrder; /* If defined an aSortOrder[i] is true, sort DESC */ + u8 *aSortOrder; /* Sort order for each column. May be NULL */ CollSeq *aColl[1]; /* Collating sequence for each term of the key */ }; /* ** An instance of the following structure holds information about a @@ -10307,11 +10570,10 @@ /* ** Internal function prototypes */ SQLITE_PRIVATE int sqlite3StrICmp(const char *, const char *); -SQLITE_PRIVATE int sqlite3IsNumber(const char*, int*, u8); SQLITE_PRIVATE int sqlite3Strlen30(const char*); #define sqlite3StrNICmp sqlite3_strnicmp SQLITE_PRIVATE int sqlite3MallocInit(void); SQLITE_PRIVATE void sqlite3MallocEnd(void); @@ -10331,11 +10593,11 @@ SQLITE_PRIVATE void sqlite3ScratchFree(void*); SQLITE_PRIVATE void *sqlite3PageMalloc(int); SQLITE_PRIVATE void sqlite3PageFree(void*); SQLITE_PRIVATE void sqlite3MemSetDefault(void); SQLITE_PRIVATE void sqlite3BenignMallocHooks(void (*)(void), void (*)(void)); -SQLITE_PRIVATE int sqlite3MemoryAlarm(void (*)(void*, sqlite3_int64, int), void*, sqlite3_int64); +SQLITE_PRIVATE int sqlite3HeapNearlyFull(void); /* ** On systems with ample stack space and that support alloca(), make ** use of alloca() to obtain space for large automatic objects. By default, ** obtain space from malloc(). @@ -10502,11 +10764,10 @@ SQLITE_PRIVATE void sqlite3ExprCachePush(Parse*); SQLITE_PRIVATE void sqlite3ExprCachePop(Parse*, int); SQLITE_PRIVATE void sqlite3ExprCacheRemove(Parse*, int, int); SQLITE_PRIVATE void sqlite3ExprCacheClear(Parse*); SQLITE_PRIVATE void sqlite3ExprCacheAffinityChange(Parse*, int, int); -SQLITE_PRIVATE void sqlite3ExprHardCopy(Parse*,int,int); SQLITE_PRIVATE int sqlite3ExprCode(Parse*, Expr*, int); SQLITE_PRIVATE int sqlite3ExprCodeTemp(Parse*, Expr*, int*); SQLITE_PRIVATE int sqlite3ExprCodeTarget(Parse*, Expr*, int); SQLITE_PRIVATE int sqlite3ExprCodeAndCache(Parse*, Expr*, int); SQLITE_PRIVATE void sqlite3ExprCodeConstants(Parse*, Expr*); @@ -10622,21 +10883,18 @@ # define sqlite3AuthContextPush(a,b,c) # define sqlite3AuthContextPop(a) ((void)(a)) #endif SQLITE_PRIVATE void sqlite3Attach(Parse*, Expr*, Expr*, Expr*); SQLITE_PRIVATE void sqlite3Detach(Parse*, Expr*); -SQLITE_PRIVATE int sqlite3BtreeFactory(sqlite3 *db, const char *zFilename, - int omitJournal, int nCache, int flags, Btree **ppBtree); SQLITE_PRIVATE int sqlite3FixInit(DbFixer*, Parse*, int, const char*, const Token*); SQLITE_PRIVATE int sqlite3FixSrcList(DbFixer*, SrcList*); SQLITE_PRIVATE int sqlite3FixSelect(DbFixer*, Select*); SQLITE_PRIVATE int sqlite3FixExpr(DbFixer*, Expr*); SQLITE_PRIVATE int sqlite3FixExprList(DbFixer*, ExprList*); SQLITE_PRIVATE int sqlite3FixTriggerStep(DbFixer*, TriggerStep*); -SQLITE_PRIVATE int sqlite3AtoF(const char *z, double*); +SQLITE_PRIVATE int sqlite3AtoF(const char *z, double*, int, u8); SQLITE_PRIVATE int sqlite3GetInt32(const char *, int*); -SQLITE_PRIVATE int sqlite3FitsIn64Bits(const char *, int); SQLITE_PRIVATE int sqlite3Utf16ByteLen(const void *pData, int nChar); SQLITE_PRIVATE int sqlite3Utf8CharLen(const char *pData, int nByte); SQLITE_PRIVATE int sqlite3Utf8Read(const u8*, const u8**); /* @@ -10678,11 +10936,11 @@ SQLITE_PRIVATE const char *sqlite3IndexAffinityStr(Vdbe *, Index *); SQLITE_PRIVATE void sqlite3TableAffinityStr(Vdbe *, Table *); SQLITE_PRIVATE char sqlite3CompareAffinity(Expr *pExpr, char aff2); SQLITE_PRIVATE int sqlite3IndexAffinityOk(Expr *pExpr, char idx_affinity); SQLITE_PRIVATE char sqlite3ExprAffinity(Expr *pExpr); -SQLITE_PRIVATE int sqlite3Atoi64(const char*, i64*); +SQLITE_PRIVATE int sqlite3Atoi64(const char*, i64*, int, u8); SQLITE_PRIVATE void sqlite3Error(sqlite3*, int, const char*,...); SQLITE_PRIVATE void *sqlite3HexToBlob(sqlite3*, const char *z, int n); SQLITE_PRIVATE int sqlite3TwoPartName(Parse *, Token *, Token *, Token **); SQLITE_PRIVATE const char *sqlite3ErrStr(int); SQLITE_PRIVATE int sqlite3ReadSchema(Parse *pParse); @@ -10749,11 +11007,13 @@ SQLITE_PRIVATE Schema *sqlite3SchemaGet(sqlite3 *, Btree *); SQLITE_PRIVATE int sqlite3SchemaToIndex(sqlite3 *db, Schema *); SQLITE_PRIVATE KeyInfo *sqlite3IndexKeyinfo(Parse *, Index *); SQLITE_PRIVATE int sqlite3CreateFunc(sqlite3 *, const char *, int, int, void *, void (*)(sqlite3_context*,int,sqlite3_value **), - void (*)(sqlite3_context*,int,sqlite3_value **), void (*)(sqlite3_context*)); + void (*)(sqlite3_context*,int,sqlite3_value **), void (*)(sqlite3_context*), + FuncDestructor *pDestructor +); SQLITE_PRIVATE int sqlite3ApiExit(sqlite3 *db, int); SQLITE_PRIVATE int sqlite3OpenTempDatabase(Parse *); SQLITE_PRIVATE void sqlite3StrAccumInit(StrAccum*, char*, int, int); SQLITE_PRIVATE void sqlite3StrAccumAppend(StrAccum*,const char*,int); @@ -11669,10 +11929,11 @@ Bool useRandomRowid; /* Generate new record numbers semi-randomly */ Bool nullRow; /* True if pointing to a row with no data */ Bool deferredMoveto; /* A call to sqlite3BtreeMoveto() is needed */ Bool isTable; /* True if a table requiring integer keys */ Bool isIndex; /* True if an index containing keys only - no data */ + Bool isOrdered; /* True if the underlying table is BTREE_UNORDERED */ i64 movetoTarget; /* Argument to the deferred sqlite3BtreeMoveto() */ Btree *pBt; /* Separate file holding temporary table */ int pseudoTableReg; /* Register holding pseudotable content. */ KeyInfo *pKeyInfo; /* Info about index keys needed by index cursors */ int nField; /* Number of fields in the header */ @@ -11763,10 +12024,14 @@ char *z; /* String or BLOB value */ int n; /* Number of characters in string value, excluding '\0' */ u16 flags; /* Some combination of MEM_Null, MEM_Str, MEM_Dyn, etc. */ u8 type; /* One of SQLITE_NULL, SQLITE_TEXT, SQLITE_INTEGER, etc */ u8 enc; /* SQLITE_UTF8, SQLITE_UTF16BE, SQLITE_UTF16LE */ +#ifdef SQLITE_DEBUG + Mem *pScopyFrom; /* This Mem is a shallow copy of pScopyFrom */ + void *pFiller; /* So that sizeof(Mem) is a multiple of 8 */ +#endif void (*xDel)(void *); /* If not null, call this function to delete Mem.z */ char *zMalloc; /* Dynamic buffer allocated by sqlite3_malloc() */ }; /* One or more of the following flags are set to indicate the validOK @@ -11789,10 +12054,11 @@ #define MEM_Int 0x0004 /* Value is an integer */ #define MEM_Real 0x0008 /* Value is a real number */ #define MEM_Blob 0x0010 /* Value is a BLOB */ #define MEM_RowSet 0x0020 /* Value is a RowSet object */ #define MEM_Frame 0x0040 /* Value is a VdbeFrame object */ +#define MEM_Invalid 0x0080 /* Value is undefined */ #define MEM_TypeMask 0x00ff /* Mask of type bits */ /* Whenever Mem contains a valid string or blob representation, one of ** the following flags must be set to determine the memory management ** policy for Mem.z. The MEM_Term flag tells us whether or not the @@ -11802,23 +12068,29 @@ #define MEM_Dyn 0x0400 /* Need to call sqliteFree() on Mem.z */ #define MEM_Static 0x0800 /* Mem.z points to a static string */ #define MEM_Ephem 0x1000 /* Mem.z points to an ephemeral string */ #define MEM_Agg 0x2000 /* Mem.z points to an agg function context */ #define MEM_Zero 0x4000 /* Mem.i contains count of 0s appended to blob */ - #ifdef SQLITE_OMIT_INCRBLOB #undef MEM_Zero #define MEM_Zero 0x0000 #endif - /* ** Clear any existing type flags from a Mem and replace them with f */ #define MemSetTypeFlag(p, f) \ ((p)->flags = ((p)->flags&~(MEM_TypeMask|MEM_Zero))|f) +/* +** Return true if a memory cell is not marked as invalid. This macro +** is for use inside assert() statements only. +*/ +#ifdef SQLITE_DEBUG +#define memIsValid(M) ((M)->flags & MEM_Invalid)==0 +#endif + /* A VdbeFunc is just a FuncDef (defined in sqliteInt.h) that contains ** additional information about auxiliary information bound to arguments ** of the function. This is used to implement the sqlite3_get_auxdata() ** and sqlite3_set_auxdata() APIs. The "auxdata" is some auxiliary data @@ -12002,10 +12274,14 @@ SQLITE_PRIVATE int sqlite3VdbeMemGrow(Mem *pMem, int n, int preserve); SQLITE_PRIVATE int sqlite3VdbeCloseStatement(Vdbe *, int); SQLITE_PRIVATE void sqlite3VdbeFrameDelete(VdbeFrame*); SQLITE_PRIVATE int sqlite3VdbeFrameRestore(VdbeFrame *); SQLITE_PRIVATE void sqlite3VdbeMemStoreType(Mem *pMem); + +#ifdef SQLITE_DEBUG +SQLITE_PRIVATE void sqlite3VdbeMemPrepareToChange(Vdbe*,Mem*); +#endif #ifndef SQLITE_OMIT_FOREIGN_KEY SQLITE_PRIVATE int sqlite3VdbeCheckFk(Vdbe *, int); #else # define sqlite3VdbeCheckFk(p,i) 0 @@ -12359,16 +12635,10 @@ end_getDigits: va_end(ap); return cnt; } -/* -** Read text from z[] and convert into a floating point number. Return -** the number of digits converted. -*/ -#define getValue sqlite3AtoF - /* ** Parse a timezone extension on the end of a date-time. ** The extension is of the form: ** ** (+/-)HH:MM @@ -12566,21 +12836,19 @@ static int parseDateOrTime( sqlite3_context *context, const char *zDate, DateTime *p ){ - int isRealNum; /* Return from sqlite3IsNumber(). Not used */ + double r; if( parseYyyyMmDd(zDate,p)==0 ){ return 0; }else if( parseHhMmSs(zDate, p)==0 ){ return 0; }else if( sqlite3StrICmp(zDate,"now")==0){ setDateTimeToCurrent(context, p); return 0; - }else if( sqlite3IsNumber(zDate, &isRealNum, SQLITE_UTF8) ){ - double r; - getValue(zDate, &r); + }else if( sqlite3AtoF(zDate, &r, sqlite3Strlen30(zDate), SQLITE_UTF8) ){ p->iJD = (sqlite3_int64)(r*86400000.0 + 0.5); p->validJD = 1; return 0; } return 1; @@ -12797,12 +13065,13 @@ ** ** Move the date to the same time on the next occurrence of ** weekday N where 0==Sunday, 1==Monday, and so forth. If the ** date is already on the appropriate weekday, this is a no-op. */ - if( strncmp(z, "weekday ", 8)==0 && getValue(&z[8],&r)>0 - && (n=(int)r)==r && n>=0 && r<7 ){ + if( strncmp(z, "weekday ", 8)==0 + && sqlite3AtoF(&z[8], &r, sqlite3Strlen30(&z[8]), SQLITE_UTF8) + && (n=(int)r)==r && n>=0 && r<7 ){ sqlite3_int64 Z; computeYMD_HMS(p); p->validTZ = 0; p->validJD = 0; computeJD(p); @@ -12853,12 +13122,15 @@ case '6': case '7': case '8': case '9': { double rRounder; - n = getValue(z, &r); - assert( n>=1 ); + for(n=1; z[n] && z[n]!=':' && !sqlite3Isspace(z[n]); n++){} + if( !sqlite3AtoF(z, &r, n, SQLITE_UTF8) ){ + rc = 1; + break; + } if( z[n]==':' ){ /* A modifier of the form (+|-)HH:MM:SS.FFF adds (or subtracts) the ** specified number of hours, minutes, seconds, and fractional seconds ** to the time. The ".FFF" may be omitted. The ":SS.FFF" may be ** omitted. @@ -13508,10 +13780,16 @@ SQLITE_PRIVATE int sqlite3OsSleep(sqlite3_vfs *pVfs, int nMicro){ return pVfs->xSleep(pVfs, nMicro); } SQLITE_PRIVATE int sqlite3OsCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *pTimeOut){ int rc; + /* IMPLEMENTATION-OF: R-49045-42493 SQLite will use the xCurrentTimeInt64() + ** method to get the current date and time if that method is available + ** (if iVersion is 2 or greater and the function pointer is not NULL) and + ** will fall back to xCurrentTime() if xCurrentTimeInt64() is + ** unavailable. + */ if( pVfs->iVersion>=2 && pVfs->xCurrentTimeInt64 ){ rc = pVfs->xCurrentTimeInt64(pVfs, pTimeOut); }else{ double r; rc = pVfs->xCurrentTime(pVfs, &r); @@ -13891,11 +14169,11 @@ ** routines and redirected to xFree. */ static void *sqlite3MemRealloc(void *pPrior, int nByte){ sqlite3_int64 *p = (sqlite3_int64*)pPrior; assert( pPrior!=0 && nByte>0 ); - nByte = ROUND8(nByte); + assert( nByte==ROUND8(nByte) ); /* EV: R-46199-30249 */ p--; p = realloc(p, nByte+8 ); if( p ){ p[0] = nByte; p++; @@ -14297,10 +14575,11 @@ */ static void *sqlite3MemRealloc(void *pPrior, int nByte){ struct MemBlockHdr *pOldHdr; void *pNew; assert( mem.disallow==0 ); + assert( (nByte & 7)==0 ); /* EV: R-46199-30249 */ pOldHdr = sqlite3MemsysGetHeader(pPrior); pNew = sqlite3MemMalloc(nByte); if( pNew ){ memcpy(pNew, pPrior, nByte<pOldHdr->iSize ? nByte : pOldHdr->iSize); if( nByte>pOldHdr->iSize ){ @@ -14331,11 +14610,11 @@ /* ** Set the "type" of an allocation. */ SQLITE_PRIVATE void sqlite3MemdebugSetType(void *p, u8 eType){ - if( p ){ + if( p && sqlite3GlobalConfig.m.xMalloc==sqlite3MemMalloc ){ struct MemBlockHdr *pHdr; pHdr = sqlite3MemsysGetHeader(p); assert( pHdr->iForeGuard==FOREGUARD ); pHdr->eType = eType; } @@ -14350,11 +14629,11 @@ ** ** assert( sqlite3MemdebugHasType(p, MEMTYPE_DB) ); */ SQLITE_PRIVATE int sqlite3MemdebugHasType(void *p, u8 eType){ int rc = 1; - if( p ){ + if( p && sqlite3GlobalConfig.m.xMalloc==sqlite3MemMalloc ){ struct MemBlockHdr *pHdr; pHdr = sqlite3MemsysGetHeader(p); assert( pHdr->iForeGuard==FOREGUARD ); /* Allocation is valid */ if( (pHdr->eType&eType)==0 ){ rc = 0; @@ -14372,11 +14651,11 @@ ** ** assert( sqlite3MemdebugNoType(p, MEMTYPE_DB) ); */ SQLITE_PRIVATE int sqlite3MemdebugNoType(void *p, u8 eType){ int rc = 1; - if( p ){ + if( p && sqlite3GlobalConfig.m.xMalloc==sqlite3MemMalloc ){ struct MemBlockHdr *pHdr; pHdr = sqlite3MemsysGetHeader(p); assert( pHdr->iForeGuard==FOREGUARD ); /* Allocation is valid */ if( (pHdr->eType&eType)!=0 ){ rc = 0; @@ -15566,11 +15845,11 @@ */ static void *memsys5Realloc(void *pPrior, int nBytes){ int nOld; void *p; assert( pPrior!=0 ); - assert( (nBytes&(nBytes-1))==0 ); + assert( (nBytes&(nBytes-1))==0 ); /* EV: R-46199-30249 */ assert( nBytes>=0 ); if( nBytes==0 ){ return 0; } nOld = memsys5Size(pPrior); @@ -16624,10 +16903,11 @@ #else /* Use the built-in recursive mutexes if they are available. */ pthread_mutex_lock(&p->mutex); #if SQLITE_MUTEX_NREF + assert( p->nRef>0 || p->owner==0 ); p->owner = pthread_self(); p->nRef++; #endif #endif @@ -16696,10 +16976,11 @@ */ static void pthreadMutexLeave(sqlite3_mutex *p){ assert( pthreadMutexHeld(p) ); #if SQLITE_MUTEX_NREF p->nRef--; + if( p->nRef==0 ) p->owner = 0; #endif assert( p->nRef==0 || p->id==SQLITE_MUTEX_RECURSIVE ); #ifdef SQLITE_HOMEGROWN_RECURSIVE_MUTEX if( p->nRef==0 ){ @@ -16960,11 +17241,11 @@ ** allocated mutex. SQLite is careful to deallocate every ** mutex that it allocates. */ static void winMutexFree(sqlite3_mutex *p){ assert( p ); - assert( p->nRef==0 ); + assert( p->nRef==0 && p->owner==0 ); assert( p->id==SQLITE_MUTEX_FAST || p->id==SQLITE_MUTEX_RECURSIVE ); DeleteCriticalSection(&p->mutex); sqlite3_free(p); } @@ -16984,10 +17265,11 @@ DWORD tid = GetCurrentThreadId(); assert( p->id==SQLITE_MUTEX_RECURSIVE || winMutexNotheld2(p, tid) ); #endif EnterCriticalSection(&p->mutex); #ifdef SQLITE_DEBUG + assert( p->nRef>0 || p->owner==0 ); p->owner = tid; p->nRef++; if( p->trace ){ printf("enter mutex %p (%d) with nRef=%d\n", p, p->trace, p->nRef); } @@ -17037,10 +17319,11 @@ #ifndef NDEBUG DWORD tid = GetCurrentThreadId(); assert( p->nRef>0 ); assert( p->owner==tid ); p->nRef--; + if( p->nRef==0 ) p->owner = 0; assert( p->nRef==0 || p->id==SQLITE_MUTEX_RECURSIVE ); #endif LeaveCriticalSection(&p->mutex); #ifdef SQLITE_DEBUG if( p->trace ){ @@ -17086,10 +17369,70 @@ ************************************************************************* ** ** Memory allocation functions used throughout sqlite. */ +/* +** Attempt to release up to n bytes of non-essential memory currently +** held by SQLite. An example of non-essential memory is memory used to +** cache database pages that are not currently in use. +*/ +SQLITE_API int sqlite3_release_memory(int n){ +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT + return sqlite3PcacheReleaseMemory(n); +#else + /* IMPLEMENTATION-OF: R-34391-24921 The sqlite3_release_memory() routine + ** is a no-op returning zero if SQLite is not compiled with + ** SQLITE_ENABLE_MEMORY_MANAGEMENT. */ + UNUSED_PARAMETER(n); + return 0; +#endif +} + +/* +** An instance of the following object records the location of +** each unused scratch buffer. +*/ +typedef struct ScratchFreeslot { + struct ScratchFreeslot *pNext; /* Next unused scratch buffer */ +} ScratchFreeslot; + +/* +** State information local to the memory allocation subsystem. +*/ +static SQLITE_WSD struct Mem0Global { + sqlite3_mutex *mutex; /* Mutex to serialize access */ + + /* + ** The alarm callback and its arguments. The mem0.mutex lock will + ** be held while the callback is running. Recursive calls into + ** the memory subsystem are allowed, but no new callbacks will be + ** issued. + */ + sqlite3_int64 alarmThreshold; + void (*alarmCallback)(void*, sqlite3_int64,int); + void *alarmArg; + + /* + ** Pointers to the end of sqlite3GlobalConfig.pScratch memory + ** (so that a range test can be used to determine if an allocation + ** being freed came from pScratch) and a pointer to the list of + ** unused scratch allocations. + */ + void *pScratchEnd; + ScratchFreeslot *pScratchFree; + u32 nScratchFree; + + /* + ** True if heap is nearly "full" where "full" is defined by the + ** sqlite3_soft_heap_limit() setting. + */ + int nearlyFull; +} mem0 = { 0, 0, 0, 0, 0, 0, 0, 0 }; + +#define mem0 GLOBAL(struct Mem0Global, mem0) + /* ** This routine runs when the memory allocator sees that the ** total memory allocation is about to exceed the soft heap ** limit. */ @@ -17100,82 +17443,70 @@ ){ UNUSED_PARAMETER2(NotUsed, NotUsed2); sqlite3_release_memory(allocSize); } +/* +** Change the alarm callback +*/ +static int sqlite3MemoryAlarm( + void(*xCallback)(void *pArg, sqlite3_int64 used,int N), + void *pArg, + sqlite3_int64 iThreshold +){ + int nUsed; + sqlite3_mutex_enter(mem0.mutex); + mem0.alarmCallback = xCallback; + mem0.alarmArg = pArg; + mem0.alarmThreshold = iThreshold; + nUsed = sqlite3StatusValue(SQLITE_STATUS_MEMORY_USED); + mem0.nearlyFull = (iThreshold>0 && iThreshold<=nUsed); + sqlite3_mutex_leave(mem0.mutex); + return SQLITE_OK; +} + +#ifndef SQLITE_OMIT_DEPRECATED +/* +** Deprecated external interface. Internal/core SQLite code +** should call sqlite3MemoryAlarm. +*/ +SQLITE_API int sqlite3_memory_alarm( + void(*xCallback)(void *pArg, sqlite3_int64 used,int N), + void *pArg, + sqlite3_int64 iThreshold +){ + return sqlite3MemoryAlarm(xCallback, pArg, iThreshold); +} +#endif + /* ** Set the soft heap-size limit for the library. Passing a zero or ** negative value indicates no limit. */ -SQLITE_API void sqlite3_soft_heap_limit(int n){ - sqlite3_uint64 iLimit; - int overage; - if( n<0 ){ - iLimit = 0; - }else{ - iLimit = n; - } +SQLITE_API sqlite3_int64 sqlite3_soft_heap_limit64(sqlite3_int64 n){ + sqlite3_int64 priorLimit; + sqlite3_int64 excess; #ifndef SQLITE_OMIT_AUTOINIT sqlite3_initialize(); #endif - if( iLimit>0 ){ - sqlite3MemoryAlarm(softHeapLimitEnforcer, 0, iLimit); + sqlite3_mutex_enter(mem0.mutex); + priorLimit = mem0.alarmThreshold; + sqlite3_mutex_leave(mem0.mutex); + if( n<0 ) return priorLimit; + if( n>0 ){ + sqlite3MemoryAlarm(softHeapLimitEnforcer, 0, n); }else{ sqlite3MemoryAlarm(0, 0, 0); } - overage = (int)(sqlite3_memory_used() - (i64)n); - if( overage>0 ){ - sqlite3_release_memory(overage); - } -} - -/* -** Attempt to release up to n bytes of non-essential memory currently -** held by SQLite. An example of non-essential memory is memory used to -** cache database pages that are not currently in use. -*/ -SQLITE_API int sqlite3_release_memory(int n){ -#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT - int nRet = 0; - nRet += sqlite3PcacheReleaseMemory(n-nRet); - return nRet; -#else - UNUSED_PARAMETER(n); - return SQLITE_OK; -#endif -} - -/* -** State information local to the memory allocation subsystem. -*/ -static SQLITE_WSD struct Mem0Global { - /* Number of free pages for scratch and page-cache memory */ - u32 nScratchFree; - u32 nPageFree; - - sqlite3_mutex *mutex; /* Mutex to serialize access */ - - /* - ** The alarm callback and its arguments. The mem0.mutex lock will - ** be held while the callback is running. Recursive calls into - ** the memory subsystem are allowed, but no new callbacks will be - ** issued. - */ - sqlite3_int64 alarmThreshold; - void (*alarmCallback)(void*, sqlite3_int64,int); - void *alarmArg; - - /* - ** Pointers to the end of sqlite3GlobalConfig.pScratch and - ** sqlite3GlobalConfig.pPage to a block of memory that records - ** which pages are available. - */ - u32 *aScratchFree; - u32 *aPageFree; -} mem0 = { 0, 0, 0, 0, 0, 0, 0, 0 }; - -#define mem0 GLOBAL(struct Mem0Global, mem0) + excess = sqlite3_memory_used() - n; + if( excess>0 ) sqlite3_release_memory((int)(excess & 0x7fffffff)); + return priorLimit; +} +SQLITE_API void sqlite3_soft_heap_limit(int n){ + if( n<0 ) n = 0; + sqlite3_soft_heap_limit64(n); +} /* ** Initialize the memory allocation subsystem. */ SQLITE_PRIVATE int sqlite3MallocInit(void){ @@ -17185,39 +17516,48 @@ memset(&mem0, 0, sizeof(mem0)); if( sqlite3GlobalConfig.bCoreMutex ){ mem0.mutex = sqlite3MutexAlloc(SQLITE_MUTEX_STATIC_MEM); } if( sqlite3GlobalConfig.pScratch && sqlite3GlobalConfig.szScratch>=100 - && sqlite3GlobalConfig.nScratch>=0 ){ - int i; - sqlite3GlobalConfig.szScratch = ROUNDDOWN8(sqlite3GlobalConfig.szScratch-4); - mem0.aScratchFree = (u32*)&((char*)sqlite3GlobalConfig.pScratch) - [sqlite3GlobalConfig.szScratch*sqlite3GlobalConfig.nScratch]; - for(i=0; i<sqlite3GlobalConfig.nScratch; i++){ mem0.aScratchFree[i] = i; } - mem0.nScratchFree = sqlite3GlobalConfig.nScratch; + && sqlite3GlobalConfig.nScratch>0 ){ + int i, n, sz; + ScratchFreeslot *pSlot; + sz = ROUNDDOWN8(sqlite3GlobalConfig.szScratch); + sqlite3GlobalConfig.szScratch = sz; + pSlot = (ScratchFreeslot*)sqlite3GlobalConfig.pScratch; + n = sqlite3GlobalConfig.nScratch; + mem0.pScratchFree = pSlot; + mem0.nScratchFree = n; + for(i=0; i<n-1; i++){ + pSlot->pNext = (ScratchFreeslot*)(sz+(char*)pSlot); + pSlot = pSlot->pNext; + } + pSlot->pNext = 0; + mem0.pScratchEnd = (void*)&pSlot[1]; }else{ + mem0.pScratchEnd = 0; sqlite3GlobalConfig.pScratch = 0; sqlite3GlobalConfig.szScratch = 0; - } - if( sqlite3GlobalConfig.pPage && sqlite3GlobalConfig.szPage>=512 - && sqlite3GlobalConfig.nPage>=1 ){ - int i; - int overhead; - int sz = ROUNDDOWN8(sqlite3GlobalConfig.szPage); - int n = sqlite3GlobalConfig.nPage; - overhead = (4*n + sz - 1)/sz; - sqlite3GlobalConfig.nPage -= overhead; - mem0.aPageFree = (u32*)&((char*)sqlite3GlobalConfig.pPage) - [sqlite3GlobalConfig.szPage*sqlite3GlobalConfig.nPage]; - for(i=0; i<sqlite3GlobalConfig.nPage; i++){ mem0.aPageFree[i] = i; } - mem0.nPageFree = sqlite3GlobalConfig.nPage; - }else{ + sqlite3GlobalConfig.nScratch = 0; + } + if( sqlite3GlobalConfig.pPage==0 || sqlite3GlobalConfig.szPage<512 + || sqlite3GlobalConfig.nPage<1 ){ sqlite3GlobalConfig.pPage = 0; sqlite3GlobalConfig.szPage = 0; + sqlite3GlobalConfig.nPage = 0; } return sqlite3GlobalConfig.m.xInit(sqlite3GlobalConfig.m.pAppData); } + +/* +** Return true if the heap is currently under memory pressure - in other +** words if the amount of heap used is close to the limit set by +** sqlite3_soft_heap_limit(). +*/ +SQLITE_PRIVATE int sqlite3HeapNearlyFull(void){ + return mem0.nearlyFull; +} /* ** Deinitialize the memory allocation subsystem. */ SQLITE_PRIVATE void sqlite3MallocEnd(void){ @@ -17249,40 +17589,10 @@ sqlite3_status(SQLITE_STATUS_MEMORY_USED, &n, &mx, resetFlag); res = (sqlite3_int64)mx; /* Work around bug in Borland C. Ticket #3216 */ return res; } -/* -** Change the alarm callback -*/ -SQLITE_PRIVATE int sqlite3MemoryAlarm( - void(*xCallback)(void *pArg, sqlite3_int64 used,int N), - void *pArg, - sqlite3_int64 iThreshold -){ - sqlite3_mutex_enter(mem0.mutex); - mem0.alarmCallback = xCallback; - mem0.alarmArg = pArg; - mem0.alarmThreshold = iThreshold; - sqlite3_mutex_leave(mem0.mutex); - return SQLITE_OK; -} - -#ifndef SQLITE_OMIT_DEPRECATED -/* -** Deprecated external interface. Internal/core SQLite code -** should call sqlite3MemoryAlarm. -*/ -SQLITE_API int sqlite3_memory_alarm( - void(*xCallback)(void *pArg, sqlite3_int64 used,int N), - void *pArg, - sqlite3_int64 iThreshold -){ - return sqlite3MemoryAlarm(xCallback, pArg, iThreshold); -} -#endif - /* ** Trigger the alarm */ static void sqlite3MallocAlarm(int nByte){ void (*xCallback)(void*,sqlite3_int64,int); @@ -17311,18 +17621,23 @@ nFull = sqlite3GlobalConfig.m.xRoundup(n); sqlite3StatusSet(SQLITE_STATUS_MALLOC_SIZE, n); if( mem0.alarmCallback!=0 ){ int nUsed = sqlite3StatusValue(SQLITE_STATUS_MEMORY_USED); if( nUsed+nFull >= mem0.alarmThreshold ){ + mem0.nearlyFull = 1; sqlite3MallocAlarm(nFull); + }else{ + mem0.nearlyFull = 0; } } p = sqlite3GlobalConfig.m.xMalloc(nFull); +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT if( p==0 && mem0.alarmCallback ){ sqlite3MallocAlarm(nFull); p = sqlite3GlobalConfig.m.xMalloc(nFull); } +#endif if( p ){ nFull = sqlite3MallocSize(p); sqlite3StatusAdd(SQLITE_STATUS_MEMORY_USED, nFull); sqlite3StatusAdd(SQLITE_STATUS_MALLOC_COUNT, 1); } @@ -17334,11 +17649,13 @@ ** Allocate memory. This routine is like sqlite3_malloc() except that it ** assumes the memory subsystem has already been initialized. */ SQLITE_PRIVATE void *sqlite3Malloc(int n){ void *p; - if( n<=0 || n>=0x7fffff00 ){ + if( n<=0 /* IMP: R-65312-04917 */ + || n>=0x7fffff00 + ){ /* A memory allocation of a number of bytes which is near the maximum ** signed integer value might cause an integer overflow inside of the ** xMalloc(). Hence we limit the maximum size to 0x7fffff00, giving ** 255 bytes of overhead. SQLite itself will never use anything near ** this amount. The only way to reach the limit is with sqlite3_malloc() */ @@ -17348,10 +17665,11 @@ mallocWithAlarm(n, &p); sqlite3_mutex_leave(mem0.mutex); }else{ p = sqlite3GlobalConfig.m.xMalloc(n); } + assert( EIGHT_BYTE_ALIGNMENT(p) ); /* IMP: R-04675-44850 */ return p; } /* ** This version of the memory allocation is for use by the application. @@ -17385,64 +17703,70 @@ ** embedded processor. */ SQLITE_PRIVATE void *sqlite3ScratchMalloc(int n){ void *p; assert( n>0 ); + + sqlite3_mutex_enter(mem0.mutex); + if( mem0.nScratchFree && sqlite3GlobalConfig.szScratch>=n ){ + p = mem0.pScratchFree; + mem0.pScratchFree = mem0.pScratchFree->pNext; + mem0.nScratchFree--; + sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_USED, 1); + sqlite3StatusSet(SQLITE_STATUS_SCRATCH_SIZE, n); + sqlite3_mutex_leave(mem0.mutex); + }else{ + if( sqlite3GlobalConfig.bMemstat ){ + sqlite3StatusSet(SQLITE_STATUS_SCRATCH_SIZE, n); + n = mallocWithAlarm(n, &p); + if( p ) sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_OVERFLOW, n); + sqlite3_mutex_leave(mem0.mutex); + }else{ + sqlite3_mutex_leave(mem0.mutex); + p = sqlite3GlobalConfig.m.xMalloc(n); + } + sqlite3MemdebugSetType(p, MEMTYPE_SCRATCH); + } + assert( sqlite3_mutex_notheld(mem0.mutex) ); + #if SQLITE_THREADSAFE==0 && !defined(NDEBUG) - /* Verify that no more than two scratch allocation per thread - ** is outstanding at one time. (This is only checked in the + /* Verify that no more than two scratch allocations per thread + ** are outstanding at one time. (This is only checked in the ** single-threaded case since checking in the multi-threaded case ** would be much more complicated.) */ assert( scratchAllocOut<=1 ); -#endif - - if( sqlite3GlobalConfig.szScratch<n ){ - goto scratch_overflow; - }else{ - sqlite3_mutex_enter(mem0.mutex); - if( mem0.nScratchFree==0 ){ - sqlite3_mutex_leave(mem0.mutex); - goto scratch_overflow; - }else{ - int i; - i = mem0.aScratchFree[--mem0.nScratchFree]; - i *= sqlite3GlobalConfig.szScratch; - sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_USED, 1); - sqlite3StatusSet(SQLITE_STATUS_SCRATCH_SIZE, n); - sqlite3_mutex_leave(mem0.mutex); - p = (void*)&((char*)sqlite3GlobalConfig.pScratch)[i]; - assert( (((u8*)p - (u8*)0) & 7)==0 ); - } - } -#if SQLITE_THREADSAFE==0 && !defined(NDEBUG) - scratchAllocOut = p!=0; -#endif - - return p; - -scratch_overflow: - if( sqlite3GlobalConfig.bMemstat ){ - sqlite3_mutex_enter(mem0.mutex); - sqlite3StatusSet(SQLITE_STATUS_SCRATCH_SIZE, n); - n = mallocWithAlarm(n, &p); - if( p ) sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_OVERFLOW, n); - sqlite3_mutex_leave(mem0.mutex); - }else{ - p = sqlite3GlobalConfig.m.xMalloc(n); - } - sqlite3MemdebugSetType(p, MEMTYPE_SCRATCH); -#if SQLITE_THREADSAFE==0 && !defined(NDEBUG) - scratchAllocOut = p!=0; -#endif - return p; + if( p ) scratchAllocOut++; +#endif + + return p; } SQLITE_PRIVATE void sqlite3ScratchFree(void *p){ if( p ){ - if( sqlite3GlobalConfig.pScratch==0 - || p<sqlite3GlobalConfig.pScratch - || p>=(void*)mem0.aScratchFree ){ + +#if SQLITE_THREADSAFE==0 && !defined(NDEBUG) + /* Verify that no more than two scratch allocation per thread + ** is outstanding at one time. (This is only checked in the + ** single-threaded case since checking in the multi-threaded case + ** would be much more complicated.) */ + assert( scratchAllocOut>=1 && scratchAllocOut<=2 ); + scratchAllocOut--; +#endif + + if( p>=sqlite3GlobalConfig.pScratch && p<mem0.pScratchEnd ){ + /* Release memory from the SQLITE_CONFIG_SCRATCH allocation */ + ScratchFreeslot *pSlot; + pSlot = (ScratchFreeslot*)p; + sqlite3_mutex_enter(mem0.mutex); + pSlot->pNext = mem0.pScratchFree; + mem0.pScratchFree = pSlot; + mem0.nScratchFree++; + assert( mem0.nScratchFree<=sqlite3GlobalConfig.nScratch ); + sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_USED, -1); + sqlite3_mutex_leave(mem0.mutex); + }else{ + /* Release memory back to the heap */ assert( sqlite3MemdebugHasType(p, MEMTYPE_SCRATCH) ); assert( sqlite3MemdebugNoType(p, ~MEMTYPE_SCRATCH) ); sqlite3MemdebugSetType(p, MEMTYPE_HEAP); if( sqlite3GlobalConfig.bMemstat ){ int iSize = sqlite3MallocSize(p); @@ -17453,30 +17777,10 @@ sqlite3GlobalConfig.m.xFree(p); sqlite3_mutex_leave(mem0.mutex); }else{ sqlite3GlobalConfig.m.xFree(p); } - }else{ - int i; - i = (int)((u8*)p - (u8*)sqlite3GlobalConfig.pScratch); - i /= sqlite3GlobalConfig.szScratch; - assert( i>=0 && i<sqlite3GlobalConfig.nScratch ); - sqlite3_mutex_enter(mem0.mutex); - assert( mem0.nScratchFree<(u32)sqlite3GlobalConfig.nScratch ); - mem0.aScratchFree[mem0.nScratchFree++] = i; - sqlite3StatusAdd(SQLITE_STATUS_SCRATCH_USED, -1); - sqlite3_mutex_leave(mem0.mutex); - -#if SQLITE_THREADSAFE==0 && !defined(NDEBUG) - /* Verify that no more than two scratch allocation per thread - ** is outstanding at one time. (This is only checked in the - ** single-threaded case since checking in the multi-threaded case - ** would be much more complicated.) */ - assert( scratchAllocOut>=1 && scratchAllocOut<=2 ); - scratchAllocOut = 0; -#endif - } } } /* @@ -17513,11 +17817,11 @@ /* ** Free memory previously obtained from sqlite3Malloc(). */ SQLITE_API void sqlite3_free(void *p){ - if( p==0 ) return; + if( p==0 ) return; /* IMP: R-49053-54554 */ assert( sqlite3MemdebugNoType(p, MEMTYPE_DB) ); assert( sqlite3MemdebugHasType(p, MEMTYPE_HEAP) ); if( sqlite3GlobalConfig.bMemstat ){ sqlite3_mutex_enter(mem0.mutex); sqlite3StatusAdd(SQLITE_STATUS_MEMORY_USED, -sqlite3MallocSize(p)); @@ -17560,21 +17864,24 @@ */ SQLITE_PRIVATE void *sqlite3Realloc(void *pOld, int nBytes){ int nOld, nNew; void *pNew; if( pOld==0 ){ - return sqlite3Malloc(nBytes); + return sqlite3Malloc(nBytes); /* IMP: R-28354-25769 */ } if( nBytes<=0 ){ - sqlite3_free(pOld); + sqlite3_free(pOld); /* IMP: R-31593-10574 */ return 0; } if( nBytes>=0x7fffff00 ){ /* The 0x7ffff00 limit term is explained in comments on sqlite3Malloc() */ return 0; } nOld = sqlite3MallocSize(pOld); + /* IMPLEMENTATION-OF: R-46199-30249 SQLite guarantees that the second + ** argument to xRealloc is always a value returned by a prior call to + ** xRoundup. */ nNew = sqlite3GlobalConfig.m.xRoundup(nBytes); if( nOld==nNew ){ pNew = pOld; }else if( sqlite3GlobalConfig.bMemstat ){ sqlite3_mutex_enter(mem0.mutex); @@ -17596,10 +17903,11 @@ } sqlite3_mutex_leave(mem0.mutex); }else{ pNew = sqlite3GlobalConfig.m.xRealloc(pOld, nNew); } + assert( EIGHT_BYTE_ALIGNMENT(pNew) ); /* IMP: R-04675-44850 */ return pNew; } /* ** The public interface to sqlite3Realloc. Make sure that the memory @@ -19759,10 +20067,16 @@ #define UpperToLower sqlite3UpperToLower /* ** Some systems have stricmp(). Others have strcasecmp(). Because ** there is no consistency, we will define our own. +** +** IMPLEMENTATION-OF: R-20522-24639 The sqlite3_strnicmp() API allows +** applications and extensions to compare the contents of two buffers +** containing UTF-8 strings in a case-independent fashion, using the same +** definition of case independence that SQLite uses internally when +** comparing identifiers. */ SQLITE_PRIVATE int sqlite3StrICmp(const char *zLeft, const char *zRight){ register unsigned char *a, *b; a = (unsigned char *)zLeft; b = (unsigned char *)zRight; @@ -19776,125 +20090,115 @@ while( N-- > 0 && *a!=0 && UpperToLower[*a]==UpperToLower[*b]){ a++; b++; } return N<0 ? 0 : UpperToLower[*a] - UpperToLower[*b]; } /* -** Return TRUE if z is a pure numeric string. Return FALSE and leave -** *realnum unchanged if the string contains any character which is not -** part of a number. -** -** If the string is pure numeric, set *realnum to TRUE if the string -** contains the '.' character or an "E+000" style exponentiation suffix. -** Otherwise set *realnum to FALSE. Note that just becaue *realnum is -** false does not mean that the number can be successfully converted into -** an integer - it might be too big. -** -** An empty string is considered non-numeric. -*/ -SQLITE_PRIVATE int sqlite3IsNumber(const char *z, int *realnum, u8 enc){ +** The string z[] is an text representation of a real number. +** Convert this string to a double and write it into *pResult. +** +** The string z[] is length bytes in length (bytes, not characters) and +** uses the encoding enc. The string is not necessarily zero-terminated. +** +** Return TRUE if the result is a valid real number (or integer) and FALSE +** if the string is empty or contains extraneous text. Valid numbers +** are in one of these formats: +** +** [+-]digits[E[+-]digits] +** [+-]digits.[digits][E[+-]digits] +** [+-].digits[E[+-]digits] +** +** Leading and trailing whitespace is ignored for the purpose of determining +** validity. +** +** If some prefix of the input string is a valid number, this routine +** returns FALSE but it still converts the prefix and writes the result +** into *pResult. +*/ +SQLITE_PRIVATE int sqlite3AtoF(const char *z, double *pResult, int length, u8 enc){ +#ifndef SQLITE_OMIT_FLOATING_POINT int incr = (enc==SQLITE_UTF8?1:2); - if( enc==SQLITE_UTF16BE ) z++; - if( *z=='-' || *z=='+' ) z += incr; - if( !sqlite3Isdigit(*z) ){ - return 0; - } - z += incr; - *realnum = 0; - while( sqlite3Isdigit(*z) ){ z += incr; } -#ifndef SQLITE_OMIT_FLOATING_POINT - if( *z=='.' ){ - z += incr; - if( !sqlite3Isdigit(*z) ) return 0; - while( sqlite3Isdigit(*z) ){ z += incr; } - *realnum = 1; - } - if( *z=='e' || *z=='E' ){ - z += incr; - if( *z=='+' || *z=='-' ) z += incr; - if( !sqlite3Isdigit(*z) ) return 0; - while( sqlite3Isdigit(*z) ){ z += incr; } - *realnum = 1; - } -#endif - return *z==0; -} - -/* -** The string z[] is an ASCII representation of a real number. -** Convert this string to a double. -** -** This routine assumes that z[] really is a valid number. If it -** is not, the result is undefined. -** -** This routine is used instead of the library atof() function because -** the library atof() might want to use "," as the decimal point instead -** of "." depending on how locale is set. But that would cause problems -** for SQL. So this routine always uses "." regardless of locale. -*/ -SQLITE_PRIVATE int sqlite3AtoF(const char *z, double *pResult){ -#ifndef SQLITE_OMIT_FLOATING_POINT - const char *zBegin = z; + const char *zEnd = z + length; /* sign * significand * (10 ^ (esign * exponent)) */ - int sign = 1; /* sign of significand */ - i64 s = 0; /* significand */ - int d = 0; /* adjust exponent for shifting decimal point */ - int esign = 1; /* sign of exponent */ - int e = 0; /* exponent */ + int sign = 1; /* sign of significand */ + i64 s = 0; /* significand */ + int d = 0; /* adjust exponent for shifting decimal point */ + int esign = 1; /* sign of exponent */ + int e = 0; /* exponent */ + int eValid = 1; /* True exponent is either not used or is well-formed */ double result; int nDigits = 0; + *pResult = 0.0; /* Default return value, in case of an error */ + + if( enc==SQLITE_UTF16BE ) z++; + /* skip leading spaces */ - while( sqlite3Isspace(*z) ) z++; + while( z<zEnd && sqlite3Isspace(*z) ) z+=incr; + if( z>=zEnd ) return 0; + /* get sign of significand */ if( *z=='-' ){ sign = -1; - z++; + z+=incr; }else if( *z=='+' ){ - z++; + z+=incr; } + /* skip leading zeroes */ - while( z[0]=='0' ) z++, nDigits++; + while( z<zEnd && z[0]=='0' ) z+=incr, nDigits++; /* copy max significant digits to significand */ - while( sqlite3Isdigit(*z) && s<((LARGEST_INT64-9)/10) ){ + while( z<zEnd && sqlite3Isdigit(*z) && s<((LARGEST_INT64-9)/10) ){ s = s*10 + (*z - '0'); - z++, nDigits++; + z+=incr, nDigits++; } + /* skip non-significant significand digits ** (increase exponent by d to shift decimal left) */ - while( sqlite3Isdigit(*z) ) z++, nDigits++, d++; + while( z<zEnd && sqlite3Isdigit(*z) ) z+=incr, nDigits++, d++; + if( z>=zEnd ) goto do_atof_calc; /* if decimal point is present */ if( *z=='.' ){ - z++; + z+=incr; /* copy digits from after decimal to significand ** (decrease exponent by d to shift decimal right) */ - while( sqlite3Isdigit(*z) && s<((LARGEST_INT64-9)/10) ){ + while( z<zEnd && sqlite3Isdigit(*z) && s<((LARGEST_INT64-9)/10) ){ s = s*10 + (*z - '0'); - z++, nDigits++, d--; + z+=incr, nDigits++, d--; } /* skip non-significant digits */ - while( sqlite3Isdigit(*z) ) z++, nDigits++; + while( z<zEnd && sqlite3Isdigit(*z) ) z+=incr, nDigits++; } + if( z>=zEnd ) goto do_atof_calc; /* if exponent is present */ if( *z=='e' || *z=='E' ){ - z++; + z+=incr; + eValid = 0; + if( z>=zEnd ) goto do_atof_calc; /* get sign of exponent */ if( *z=='-' ){ esign = -1; - z++; + z+=incr; }else if( *z=='+' ){ - z++; + z+=incr; } /* copy digits to exponent */ - while( sqlite3Isdigit(*z) ){ + while( z<zEnd && sqlite3Isdigit(*z) ){ e = e*10 + (*z - '0'); - z++; + z+=incr; + eValid = 1; } } + /* skip trailing spaces */ + if( nDigits && eValid ){ + while( z<zEnd && sqlite3Isspace(*z) ) z+=incr; + } + +do_atof_calc: /* adjust exponent by d, and update sign */ e = (e*esign) + d; if( e<0 ) { esign = -1; e *= -1; @@ -19949,132 +20253,104 @@ } /* store the result */ *pResult = result; - /* return number of characters used */ - return (int)(z - zBegin); + /* return true if number and no extra non-whitespace chracters after */ + return z>=zEnd && nDigits>0 && eValid; #else - return sqlite3Atoi64(z, pResult); + return !sqlite3Atoi64(z, pResult, length, enc); #endif /* SQLITE_OMIT_FLOATING_POINT */ } /* ** Compare the 19-character string zNum against the text representation ** value 2^63: 9223372036854775808. Return negative, zero, or positive ** if zNum is less than, equal to, or greater than the string. +** Note that zNum must contain exactly 19 characters. ** ** Unlike memcmp() this routine is guaranteed to return the difference ** in the values of the last digit if the only difference is in the ** last digit. So, for example, ** -** compare2pow63("9223372036854775800") +** compare2pow63("9223372036854775800", 1) ** ** will return -8. */ -static int compare2pow63(const char *zNum){ - int c; - c = memcmp(zNum,"922337203685477580",18)*10; +static int compare2pow63(const char *zNum, int incr){ + int c = 0; + int i; + /* 012345678901234567 */ + const char *pow63 = "922337203685477580"; + for(i=0; c==0 && i<18; i++){ + c = (zNum[i*incr]-pow63[i])*10; + } if( c==0 ){ - c = zNum[18] - '8'; + c = zNum[18*incr] - '8'; testcase( c==(-1) ); testcase( c==0 ); testcase( c==(+1) ); } return c; } /* -** Return TRUE if zNum is a 64-bit signed integer and write -** the value of the integer into *pNum. If zNum is not an integer -** or is an integer that is too large to be expressed with 64 bits, -** then return false. +** Convert zNum to a 64-bit signed integer and write +** the value of the integer into *pNum. +** If zNum is exactly 9223372036854665808, return 2. +** This is a special case as the context will determine +** if it is too big (used as a negative). +** If zNum is not an integer or is an integer that +** is too large to be expressed with 64 bits, +** then return 1. Otherwise return 0. ** -** When this routine was originally written it dealt with only -** 32-bit numbers. At that time, it was much faster than the -** atoi() library routine in RedHat 7.2. +** length is the number of bytes in the string (bytes, not characters). +** The string is not necessarily zero-terminated. The encoding is +** given by enc. */ -SQLITE_PRIVATE int sqlite3Atoi64(const char *zNum, i64 *pNum){ +SQLITE_PRIVATE int sqlite3Atoi64(const char *zNum, i64 *pNum, int length, u8 enc){ + int incr = (enc==SQLITE_UTF8?1:2); i64 v = 0; - int neg; - int i, c; + int neg = 0; /* assume positive */ + int i; + int c = 0; const char *zStart; - while( sqlite3Isspace(*zNum) ) zNum++; + const char *zEnd = zNum + length; + if( enc==SQLITE_UTF16BE ) zNum++; + while( zNum<zEnd && sqlite3Isspace(*zNum) ) zNum+=incr; + if( zNum>=zEnd ) goto do_atoi_calc; if( *zNum=='-' ){ neg = 1; - zNum++; + zNum+=incr; }else if( *zNum=='+' ){ - neg = 0; - zNum++; - }else{ - neg = 0; + zNum+=incr; } +do_atoi_calc: zStart = zNum; - while( zNum[0]=='0' ){ zNum++; } /* Skip over leading zeros. Ticket #2454 */ - for(i=0; (c=zNum[i])>='0' && c<='9'; i++){ + while( zNum<zEnd && zNum[0]=='0' ){ zNum+=incr; } /* Skip leading zeros. */ + for(i=0; &zNum[i]<zEnd && (c=zNum[i])>='0' && c<='9'; i+=incr){ v = v*10 + c - '0'; } *pNum = neg ? -v : v; testcase( i==18 ); testcase( i==19 ); testcase( i==20 ); - if( c!=0 || (i==0 && zStart==zNum) || i>19 ){ + if( (c!=0 && &zNum[i]<zEnd) || (i==0 && zStart==zNum) || i>19*incr ){ /* zNum is empty or contains non-numeric text or is longer - ** than 19 digits (thus guaranting that it is too large) */ - return 0; - }else if( i<19 ){ + ** than 19 digits (thus guaranteeing that it is too large) */ + return 1; + }else if( i<19*incr ){ /* Less than 19 digits, so we know that it fits in 64 bits */ - return 1; + return 0; }else{ /* 19-digit numbers must be no larger than 9223372036854775807 if positive ** or 9223372036854775808 if negative. Note that 9223372036854665808 - ** is 2^63. */ - return compare2pow63(zNum)<neg; - } -} - -/* -** The string zNum represents an unsigned integer. The zNum string -** consists of one or more digit characters and is terminated by -** a zero character. Any stray characters in zNum result in undefined -** behavior. -** -** If the unsigned integer that zNum represents will fit in a -** 64-bit signed integer, return TRUE. Otherwise return FALSE. -** -** If the negFlag parameter is true, that means that zNum really represents -** a negative number. (The leading "-" is omitted from zNum.) This -** parameter is needed to determine a boundary case. A string -** of "9223373036854775808" returns false if negFlag is false or true -** if negFlag is true. -** -** Leading zeros are ignored. -*/ -SQLITE_PRIVATE int sqlite3FitsIn64Bits(const char *zNum, int negFlag){ - int i; - int neg = 0; - - assert( zNum[0]>='0' && zNum[0]<='9' ); /* zNum is an unsigned number */ - - if( negFlag ) neg = 1-neg; - while( *zNum=='0' ){ - zNum++; /* Skip leading zeros. Ticket #2454 */ - } - for(i=0; zNum[i]; i++){ assert( zNum[i]>='0' && zNum[i]<='9' ); } - testcase( i==18 ); - testcase( i==19 ); - testcase( i==20 ); - if( i<19 ){ - /* Guaranteed to fit if less than 19 digits */ - return 1; - }else if( i>19 ){ - /* Guaranteed to be too big if greater than 19 digits */ - return 0; - }else{ - /* Compare against 2^63. */ - return compare2pow63(zNum)<neg; + ** is 2^63. Return 1 if to large */ + c=compare2pow63(zNum, incr); + if( c==0 && neg==0 ) return 2; /* too big, exactly 9223372036854665808 */ + return c<neg ? 0 : 1; } } /* ** If zNum represents an integer that will fit in 32-bits, then set @@ -22609,10 +22885,11 @@ void *lockingContext; /* Locking style specific state */ UnixUnusedFd *pUnused; /* Pre-allocated UnixUnusedFd */ int fileFlags; /* Miscellanous flags */ const char *zPath; /* Name of the file */ unixShm *pShm; /* Shared memory segment information */ + int szChunk; /* Configured by FCNTL_CHUNK_SIZE */ #if SQLITE_ENABLE_LOCKING_STYLE int openFlags; /* The flags specified at open() */ #endif #if SQLITE_ENABLE_LOCKING_STYLE || defined(__APPLE__) unsigned fsFlags; /* cached details from statfs() */ @@ -25367,19 +25644,21 @@ offset += wrote; pBuf = &((char*)pBuf)[wrote]; } SimulateIOError(( wrote=(-1), amt=1 )); SimulateDiskfullError(( wrote=0, amt=1 )); + if( amt>0 ){ if( wrote<0 ){ /* lastErrno set by seekAndWrite */ return SQLITE_IOERR_WRITE; }else{ pFile->lastErrno = 0; /* not a system error */ return SQLITE_FULL; } } + return SQLITE_OK; } #ifdef SQLITE_TEST /* @@ -25577,16 +25856,27 @@ /* ** Truncate an open file to a specified size */ static int unixTruncate(sqlite3_file *id, i64 nByte){ + unixFile *pFile = (unixFile *)id; int rc; - assert( id ); + assert( pFile ); SimulateIOError( return SQLITE_IOERR_TRUNCATE ); - rc = ftruncate(((unixFile*)id)->h, (off_t)nByte); + + /* If the user has configured a chunk-size for this file, truncate the + ** file so that it consists of an integer number of chunks (i.e. the + ** actual file size after the operation may be larger than the requested + ** size). + */ + if( pFile->szChunk ){ + nByte = ((nByte + pFile->szChunk - 1)/pFile->szChunk) * pFile->szChunk; + } + + rc = ftruncate(pFile->h, (off_t)nByte); if( rc ){ - ((unixFile*)id)->lastErrno = errno; + pFile->lastErrno = errno; return SQLITE_IOERR_TRUNCATE; }else{ #ifndef NDEBUG /* If we are doing a normal write to a database file (as opposed to ** doing a hot-journal rollback or a write to some file other than a @@ -25593,12 +25883,12 @@ ** normal database file) and we truncate the file to zero length, ** that effectively updates the change counter. This might happen ** when restoring a database using the backup API from a zero-length ** source. */ - if( ((unixFile*)id)->inNormalWrite && nByte==0 ){ - ((unixFile*)id)->transCntrChng = 1; + if( pFile->inNormalWrite && nByte==0 ){ + pFile->transCntrChng = 1; } #endif return SQLITE_OK; } @@ -25637,10 +25927,58 @@ ** proxying locking division. */ static int proxyFileControl(sqlite3_file*,int,void*); #endif +/* +** This function is called to handle the SQLITE_FCNTL_SIZE_HINT +** file-control operation. +** +** If the user has configured a chunk-size for this file, it could be +** that the file needs to be extended at this point. Otherwise, the +** SQLITE_FCNTL_SIZE_HINT operation is a no-op for Unix. +*/ +static int fcntlSizeHint(unixFile *pFile, i64 nByte){ + if( pFile->szChunk ){ + i64 nSize; /* Required file size */ + struct stat buf; /* Used to hold return values of fstat() */ + + if( fstat(pFile->h, &buf) ) return SQLITE_IOERR_FSTAT; + + nSize = ((nByte+pFile->szChunk-1) / pFile->szChunk) * pFile->szChunk; + if( nSize>(i64)buf.st_size ){ +#if defined(HAVE_POSIX_FALLOCATE) && HAVE_POSIX_FALLOCATE + if( posix_fallocate(pFile->h, buf.st_size, nSize-buf.st_size) ){ + return SQLITE_IOERR_WRITE; + } +#else + /* If the OS does not have posix_fallocate(), fake it. First use + ** ftruncate() to set the file size, then write a single byte to + ** the last byte in each block within the extended region. This + ** is the same technique used by glibc to implement posix_fallocate() + ** on systems that do not have a real fallocate() system call. + */ + int nBlk = buf.st_blksize; /* File-system block size */ + i64 iWrite; /* Next offset to write to */ + int nWrite; /* Return value from seekAndWrite() */ + + if( ftruncate(pFile->h, nSize) ){ + pFile->lastErrno = errno; + return SQLITE_IOERR_TRUNCATE; + } + iWrite = ((buf.st_size + 2*nBlk - 1)/nBlk)*nBlk-1; + do { + nWrite = seekAndWrite(pFile, iWrite, "", 1); + iWrite += nBlk; + } while( nWrite==1 && iWrite<nSize ); + if( nWrite!=1 ) return SQLITE_IOERR_WRITE; +#endif + } + } + + return SQLITE_OK; +} /* ** Information and control of an open file handle. */ static int unixFileControl(sqlite3_file *id, int op, void *pArg){ @@ -25650,18 +25988,17 @@ return SQLITE_OK; } case SQLITE_LAST_ERRNO: { *(int*)pArg = ((unixFile*)id)->lastErrno; return SQLITE_OK; + } + case SQLITE_FCNTL_CHUNK_SIZE: { + ((unixFile*)id)->szChunk = *(int *)pArg; + return SQLITE_OK; } case SQLITE_FCNTL_SIZE_HINT: { -#if 0 /* No performance advantage seen on Linux */ - sqlite3_int64 szFile = *(sqlite3_int64*)pArg; - unixFile *pFile = (unixFile*)id; - ftruncate(pFile->h, szFile); -#endif - return SQLITE_OK; + return fcntlSizeHint((unixFile *)id, *(i64 *)pArg); } #ifndef NDEBUG /* The pager calls this method to signal that it has done ** a rollback and that the database is therefore unchanged and ** it hence it is OK for the transaction change counter to be @@ -26102,11 +26439,11 @@ goto shmpage_out; } pShmNode->apRegion = apNew; while(pShmNode->nRegion<=iRegion){ void *pMem = mmap(0, szRegion, PROT_READ|PROT_WRITE, - MAP_SHARED, pShmNode->h, iRegion*szRegion + MAP_SHARED, pShmNode->h, pShmNode->nRegion*szRegion ); if( pMem==MAP_FAILED ){ rc = SQLITE_IOERR; goto shmpage_out; } @@ -26619,16 +26956,26 @@ /* Parameter isDelete is only used on vxworks. Express this explicitly ** here to prevent compiler warnings about unused parameters. */ UNUSED_PARAMETER(isDelete); + + /* Usually the path zFilename should not be a relative pathname. The + ** exception is when opening the proxy "conch" file in builds that + ** include the special Apple locking styles. + */ +#if defined(__APPLE__) && SQLITE_ENABLE_LOCKING_STYLE + assert( zFilename==0 || zFilename[0]=='/' + || pVfs->pAppData==(void*)&autolockIoFinder ); +#else + assert( zFilename==0 || zFilename[0]=='/' ); +#endif OSTRACE(("OPEN %-3d %s\n", h, zFilename)); pNew->h = h; pNew->dirfd = dirfd; pNew->fileFlags = 0; - assert( zFilename==0 || zFilename[0]=='/' ); /* Never a relative pathname */ pNew->zPath = zFilename; #if OS_VXWORKS pNew->pId = vxworksFindFileId(zFilename); if( pNew->pId==0 ){ @@ -27925,17 +28272,20 @@ static int proxyGetHostID(unsigned char *pHostID, int *pError){ struct timespec timeout = {1, 0}; /* 1 sec timeout */ assert(PROXY_HOSTIDLEN == sizeof(uuid_t)); memset(pHostID, 0, PROXY_HOSTIDLEN); +#if defined(__MAX_OS_X_VERSION_MIN_REQUIRED)\ + && __MAC_OS_X_VERSION_MIN_REQUIRED<1050 if( gethostuuid(pHostID, &timeout) ){ int err = errno; if( pError ){ *pError = err; } return SQLITE_IOERR; } +#endif #ifdef SQLITE_TEST /* simulate multiple hosts by creating unique hostid file paths */ if( sqlite3_hostid_num != 0){ pHostID[0] = (char)(pHostID[0] + (char)(sqlite3_hostid_num & 0xFF)); } @@ -28522,11 +28872,11 @@ }else{ if( pCtx->conchFile ){ pCtx->conchFile->pMethod->xClose((sqlite3_file *)pCtx->conchFile); sqlite3_free(pCtx->conchFile); } - sqlite3_free(pCtx->lockProxyPath); + sqlite3DbFree(0, pCtx->lockProxyPath); sqlite3_free(pCtx->conchFilePath); sqlite3_free(pCtx); } OSTRACE(("TRANSPROXY %d %s\n", pFile->h, (rc==SQLITE_OK ? "ok" : "failed"))); @@ -28713,13 +29063,13 @@ } rc = conchFile->pMethod->xClose((sqlite3_file*)conchFile); if( rc ) return rc; sqlite3_free(conchFile); } - sqlite3_free(pCtx->lockProxyPath); + sqlite3DbFree(0, pCtx->lockProxyPath); sqlite3_free(pCtx->conchFilePath); - sqlite3_free(pCtx->dbPath); + sqlite3DbFree(0, pCtx->dbPath); /* restore the original locking context and pMethod then close it */ pFile->lockingContext = pCtx->oldLockingContext; pFile->pMethod = pCtx->pOldMethod; sqlite3_free(pCtx); return pFile->pMethod->xClose(id); @@ -29161,10 +29511,11 @@ short sharedLockByte; /* Randomly chosen byte used as a shared lock */ DWORD lastErrno; /* The Windows errno from the last I/O error */ DWORD sectorSize; /* Sector size of the device file is on */ winShm *pShm; /* Instance of shared memory on this file */ const char *zPath; /* Full pathname of this file */ + int szChunk; /* Chunk size configured by FCNTL_CHUNK_SIZE */ #if SQLITE_OS_WINCE WCHAR *zDeleteOnClose; /* Name of file to delete when closing */ HANDLE hMutex; /* Mutex used to control access to shared lock */ HANDLE hShared; /* Shared memory segment used for locking */ winceLock local; /* Locks obtained by this instance of winFile */ @@ -29671,10 +30022,46 @@ /***************************************************************************** ** The next group of routines implement the I/O methods specified ** by the sqlite3_io_methods object. ******************************************************************************/ + +/* +** Some microsoft compilers lack this definition. +*/ +#ifndef INVALID_SET_FILE_POINTER +# define INVALID_SET_FILE_POINTER ((DWORD)-1) +#endif + +/* +** Move the current position of the file handle passed as the first +** argument to offset iOffset within the file. If successful, return 0. +** Otherwise, set pFile->lastErrno and return non-zero. +*/ +static int seekWinFile(winFile *pFile, sqlite3_int64 iOffset){ + LONG upperBits; /* Most sig. 32 bits of new offset */ + LONG lowerBits; /* Least sig. 32 bits of new offset */ + DWORD dwRet; /* Value returned by SetFilePointer() */ + + upperBits = (LONG)((iOffset>>32) & 0x7fffffff); + lowerBits = (LONG)(iOffset & 0xffffffff); + + /* API oddity: If successful, SetFilePointer() returns a dword + ** containing the lower 32-bits of the new file-offset. Or, if it fails, + ** it returns INVALID_SET_FILE_POINTER. However according to MSDN, + ** INVALID_SET_FILE_POINTER may also be a valid new offset. So to determine + ** whether an error has actually occured, it is also necessary to call + ** GetLastError(). + */ + dwRet = SetFilePointer(pFile->h, lowerBits, &upperBits, FILE_BEGIN); + if( (dwRet==INVALID_SET_FILE_POINTER && GetLastError()!=NO_ERROR) ){ + pFile->lastErrno = GetLastError(); + return 1; + } + + return 0; +} /* ** Close a file. ** ** It is reported that an attempt to close a handle might sometimes @@ -29714,17 +30101,10 @@ OSTRACE(("CLOSE %d %s\n", pFile->h, rc ? "ok" : "failed")); OpenCounter(-1); return rc ? SQLITE_OK : SQLITE_IOERR; } -/* -** Some microsoft compilers lack this definition. -*/ -#ifndef INVALID_SET_FILE_POINTER -# define INVALID_SET_FILE_POINTER ((DWORD)-1) -#endif - /* ** Read data from a file into a buffer. Return SQLITE_OK if all ** bytes were read successfully and SQLITE_IOERR if anything goes ** wrong. */ @@ -29732,112 +30112,108 @@ sqlite3_file *id, /* File to read from */ void *pBuf, /* Write content into this buffer */ int amt, /* Number of bytes to read */ sqlite3_int64 offset /* Begin reading at this offset */ ){ - LONG upperBits = (LONG)((offset>>32) & 0x7fffffff); - LONG lowerBits = (LONG)(offset & 0xffffffff); - DWORD rc; - winFile *pFile = (winFile*)id; - DWORD error; - DWORD got; + winFile *pFile = (winFile*)id; /* file handle */ + DWORD nRead; /* Number of bytes actually read from file */ assert( id!=0 ); SimulateIOError(return SQLITE_IOERR_READ); OSTRACE(("READ %d lock=%d\n", pFile->h, pFile->locktype)); - rc = SetFilePointer(pFile->h, lowerBits, &upperBits, FILE_BEGIN); - if( rc==INVALID_SET_FILE_POINTER && (error=GetLastError())!=NO_ERROR ){ - pFile->lastErrno = error; + + if( seekWinFile(pFile, offset) ){ return SQLITE_FULL; } - if( !ReadFile(pFile->h, pBuf, amt, &got, 0) ){ + if( !ReadFile(pFile->h, pBuf, amt, &nRead, 0) ){ pFile->lastErrno = GetLastError(); return SQLITE_IOERR_READ; } - if( got==(DWORD)amt ){ - return SQLITE_OK; - }else{ + if( nRead<(DWORD)amt ){ /* Unread parts of the buffer must be zero-filled */ - memset(&((char*)pBuf)[got], 0, amt-got); + memset(&((char*)pBuf)[nRead], 0, amt-nRead); return SQLITE_IOERR_SHORT_READ; } + + return SQLITE_OK; } /* ** Write data from a buffer into a file. Return SQLITE_OK on success ** or some other error code on failure. */ static int winWrite( - sqlite3_file *id, /* File to write into */ - const void *pBuf, /* The bytes to be written */ - int amt, /* Number of bytes to write */ - sqlite3_int64 offset /* Offset into the file to begin writing at */ + sqlite3_file *id, /* File to write into */ + const void *pBuf, /* The bytes to be written */ + int amt, /* Number of bytes to write */ + sqlite3_int64 offset /* Offset into the file to begin writing at */ ){ - LONG upperBits = (LONG)((offset>>32) & 0x7fffffff); - LONG lowerBits = (LONG)(offset & 0xffffffff); - DWORD rc; - winFile *pFile = (winFile*)id; - DWORD error; - DWORD wrote = 0; - - assert( id!=0 ); + int rc; /* True if error has occured, else false */ + winFile *pFile = (winFile*)id; /* File handle */ + + assert( amt>0 ); + assert( pFile ); SimulateIOError(return SQLITE_IOERR_WRITE); SimulateDiskfullError(return SQLITE_FULL); - OSTRACE(("WRITE %d lock=%d\n", pFile->h, pFile->locktype)); - rc = SetFilePointer(pFile->h, lowerBits, &upperBits, FILE_BEGIN); - if( rc==INVALID_SET_FILE_POINTER && (error=GetLastError())!=NO_ERROR ){ - pFile->lastErrno = error; - if( pFile->lastErrno==ERROR_HANDLE_DISK_FULL ){ - return SQLITE_FULL; - }else{ - return SQLITE_IOERR_WRITE; - } - } - assert( amt>0 ); - while( - amt>0 - && (rc = WriteFile(pFile->h, pBuf, amt, &wrote, 0))!=0 - && wrote>0 - ){ - amt -= wrote; - pBuf = &((char*)pBuf)[wrote]; - } - if( !rc || amt>(int)wrote ){ - pFile->lastErrno = GetLastError(); - if( pFile->lastErrno==ERROR_HANDLE_DISK_FULL ){ - return SQLITE_FULL; - }else{ - return SQLITE_IOERR_WRITE; - } + + OSTRACE(("WRITE %d lock=%d\n", pFile->h, pFile->locktype)); + + rc = seekWinFile(pFile, offset); + if( rc==0 ){ + u8 *aRem = (u8 *)pBuf; /* Data yet to be written */ + int nRem = amt; /* Number of bytes yet to be written */ + DWORD nWrite; /* Bytes written by each WriteFile() call */ + + while( nRem>0 && WriteFile(pFile->h, aRem, nRem, &nWrite, 0) && nWrite>0 ){ + aRem += nWrite; + nRem -= nWrite; + } + if( nRem>0 ){ + pFile->lastErrno = GetLastError(); + rc = 1; + } + } + + if( rc ){ + if( pFile->lastErrno==ERROR_HANDLE_DISK_FULL ){ + return SQLITE_FULL; + } + return SQLITE_IOERR_WRITE; } return SQLITE_OK; } /* ** Truncate an open file to a specified size */ static int winTruncate(sqlite3_file *id, sqlite3_int64 nByte){ - LONG upperBits = (LONG)((nByte>>32) & 0x7fffffff); - LONG lowerBits = (LONG)(nByte & 0xffffffff); - DWORD dwRet; - winFile *pFile = (winFile*)id; - DWORD error; - int rc = SQLITE_OK; - - assert( id!=0 ); + winFile *pFile = (winFile*)id; /* File handle object */ + int rc = SQLITE_OK; /* Return code for this function */ + + assert( pFile ); + OSTRACE(("TRUNCATE %d %lld\n", pFile->h, nByte)); SimulateIOError(return SQLITE_IOERR_TRUNCATE); - dwRet = SetFilePointer(pFile->h, lowerBits, &upperBits, FILE_BEGIN); - if( dwRet==INVALID_SET_FILE_POINTER && (error=GetLastError())!=NO_ERROR ){ - pFile->lastErrno = error; + + /* If the user has configured a chunk-size for this file, truncate the + ** file so that it consists of an integer number of chunks (i.e. the + ** actual file size after the operation may be larger than the requested + ** size). + */ + if( pFile->szChunk ){ + nByte = ((nByte + pFile->szChunk - 1)/pFile->szChunk) * pFile->szChunk; + } + + /* SetEndOfFile() returns non-zero when successful, or zero when it fails. */ + if( seekWinFile(pFile, nByte) ){ rc = SQLITE_IOERR_TRUNCATE; - /* SetEndOfFile will fail if nByte is negative */ - }else if( !SetEndOfFile(pFile->h) ){ + }else if( 0==SetEndOfFile(pFile->h) ){ pFile->lastErrno = GetLastError(); rc = SQLITE_IOERR_TRUNCATE; } - OSTRACE(("TRUNCATE %d %lld %s\n", pFile->h, nByte, rc==SQLITE_OK ? "ok" : "failed")); + + OSTRACE(("TRUNCATE %d %lld %s\n", pFile->h, nByte, rc ? "failed" : "ok")); return rc; } #ifdef SQLITE_TEST /* @@ -30197,10 +30573,14 @@ return SQLITE_OK; } case SQLITE_LAST_ERRNO: { *(int*)pArg = (int)((winFile*)id)->lastErrno; return SQLITE_OK; + } + case SQLITE_FCNTL_CHUNK_SIZE: { + ((winFile*)id)->szChunk = *(int *)pArg; + return SQLITE_OK; } case SQLITE_FCNTL_SIZE_HINT: { sqlite3_int64 sz = *(sqlite3_int64*)pArg; SimulateIOErrorBenign(1); winTruncate(id, sz); @@ -30234,10 +30614,18 @@ return SQLITE_IOCAP_UNDELETABLE_WHEN_OPEN; } #ifndef SQLITE_OMIT_WAL +/* +** Windows will only let you create file view mappings +** on allocation size granularity boundaries. +** During sqlite3_os_init() we do a GetSystemInfo() +** to get the granularity size. +*/ +SYSTEM_INFO winSysInfo; + /* ** Helper functions to obtain and relinquish the global mutex. The ** global mutex is used to protect the winLockInfo objects used by ** this file, all of which may be shared by multiple threads. ** @@ -30402,19 +30790,26 @@ ** by VFS shared-memory methods. */ static void winShmPurge(sqlite3_vfs *pVfs, int deleteFlag){ winShmNode **pp; winShmNode *p; + BOOL bRc; assert( winShmMutexHeld() ); pp = &winShmNodeList; while( (p = *pp)!=0 ){ if( p->nRef==0 ){ int i; if( p->mutex ) sqlite3_mutex_free(p->mutex); for(i=0; i<p->nRegion; i++){ - UnmapViewOfFile(p->aRegion[i].pMap); - CloseHandle(p->aRegion[i].hMap); + bRc = UnmapViewOfFile(p->aRegion[i].pMap); + OSTRACE(("SHM-PURGE pid-%d unmap region=%d %s\n", + (int)GetCurrentProcessId(), i, + bRc ? "ok" : "failed")); + bRc = CloseHandle(p->aRegion[i].hMap); + OSTRACE(("SHM-PURGE pid-%d close region=%d %s\n", + (int)GetCurrentProcessId(), i, + bRc ? "ok" : "failed")); } if( p->hFile.h != INVALID_HANDLE_VALUE ){ SimulateIOErrorBenign(1); winClose((sqlite3_file *)&p->hFile); SimulateIOErrorBenign(0); @@ -30487,14 +30882,15 @@ pShmNode->mutex = sqlite3_mutex_alloc(SQLITE_MUTEX_FAST); if( pShmNode->mutex==0 ){ rc = SQLITE_NOMEM; goto shm_open_err; } + rc = winOpen(pDbFd->pVfs, pShmNode->zFilename, /* Name of the file (UTF-8) */ (sqlite3_file*)&pShmNode->hFile, /* File handle here */ - SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, /* Mode flags */ + SQLITE_OPEN_WAL | SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, /* Mode flags */ 0); if( SQLITE_OK!=rc ){ rc = SQLITE_CANTOPEN_BKPT; goto shm_open_err; } @@ -30798,14 +31194,22 @@ void *pMap = 0; /* Mapped memory region */ hMap = CreateFileMapping(pShmNode->hFile.h, NULL, PAGE_READWRITE, 0, nByte, NULL ); + OSTRACE(("SHM-MAP pid-%d create region=%d nbyte=%d %s\n", + (int)GetCurrentProcessId(), pShmNode->nRegion, nByte, + hMap ? "ok" : "failed")); if( hMap ){ + int iOffset = pShmNode->nRegion*szRegion; + int iOffsetShift = iOffset % winSysInfo.dwAllocationGranularity; pMap = MapViewOfFile(hMap, FILE_MAP_WRITE | FILE_MAP_READ, - 0, 0, nByte + 0, iOffset - iOffsetShift, szRegion + iOffsetShift ); + OSTRACE(("SHM-MAP pid-%d map region=%d offset=%d size=%d %s\n", + (int)GetCurrentProcessId(), pShmNode->nRegion, iOffset, szRegion, + pMap ? "ok" : "failed")); } if( !pMap ){ pShmNode->lastErrno = GetLastError(); rc = SQLITE_IOERR; if( hMap ) CloseHandle(hMap); @@ -30818,12 +31222,14 @@ } } shmpage_out: if( pShmNode->nRegion>iRegion ){ + int iOffset = iRegion*szRegion; + int iOffsetShift = iOffset % winSysInfo.dwAllocationGranularity; char *p = (char *)pShmNode->aRegion[iRegion].pMap; - *pp = (void *)&p[iRegion*szRegion]; + *pp = (void *)&p[iOffsetShift]; }else{ *pp = 0; } sqlite3_mutex_leave(pShmNode->mutex); return rc; @@ -31046,13 +31452,64 @@ DWORD dwFlagsAndAttributes = 0; #if SQLITE_OS_WINCE int isTemp = 0; #endif winFile *pFile = (winFile*)id; - void *zConverted; /* Filename in OS encoding */ - const char *zUtf8Name = zName; /* Filename in UTF-8 encoding */ - char zTmpname[MAX_PATH+1]; /* Buffer used to create temp filename */ + void *zConverted; /* Filename in OS encoding */ + const char *zUtf8Name = zName; /* Filename in UTF-8 encoding */ + + /* If argument zPath is a NULL pointer, this function is required to open + ** a temporary file. Use this buffer to store the file name in. + */ + char zTmpname[MAX_PATH+1]; /* Buffer used to create temp filename */ + + int rc = SQLITE_OK; /* Function Return Code */ +#if !defined(NDEBUG) || SQLITE_OS_WINCE + int eType = flags&0xFFFFFF00; /* Type of file to open */ +#endif + + int isExclusive = (flags & SQLITE_OPEN_EXCLUSIVE); + int isDelete = (flags & SQLITE_OPEN_DELETEONCLOSE); + int isCreate = (flags & SQLITE_OPEN_CREATE); +#ifndef NDEBUG + int isReadonly = (flags & SQLITE_OPEN_READONLY); +#endif + int isReadWrite = (flags & SQLITE_OPEN_READWRITE); + +#ifndef NDEBUG + int isOpenJournal = (isCreate && ( + eType==SQLITE_OPEN_MASTER_JOURNAL + || eType==SQLITE_OPEN_MAIN_JOURNAL + || eType==SQLITE_OPEN_WAL + )); +#endif + + /* Check the following statements are true: + ** + ** (a) Exactly one of the READWRITE and READONLY flags must be set, and + ** (b) if CREATE is set, then READWRITE must also be set, and + ** (c) if EXCLUSIVE is set, then CREATE must also be set. + ** (d) if DELETEONCLOSE is set, then CREATE must also be set. + */ + assert((isReadonly==0 || isReadWrite==0) && (isReadWrite || isReadonly)); + assert(isCreate==0 || isReadWrite); + assert(isExclusive==0 || isCreate); + assert(isDelete==0 || isCreate); + + /* The main DB, main journal, WAL file and master journal are never + ** automatically deleted. Nor are they ever temporary files. */ + assert( (!isDelete && zName) || eType!=SQLITE_OPEN_MAIN_DB ); + assert( (!isDelete && zName) || eType!=SQLITE_OPEN_MAIN_JOURNAL ); + assert( (!isDelete && zName) || eType!=SQLITE_OPEN_MASTER_JOURNAL ); + assert( (!isDelete && zName) || eType!=SQLITE_OPEN_WAL ); + + /* Assert that the upper layer has set one of the "file-type" flags. */ + assert( eType==SQLITE_OPEN_MAIN_DB || eType==SQLITE_OPEN_TEMP_DB + || eType==SQLITE_OPEN_MAIN_JOURNAL || eType==SQLITE_OPEN_TEMP_JOURNAL + || eType==SQLITE_OPEN_SUBJOURNAL || eType==SQLITE_OPEN_MASTER_JOURNAL + || eType==SQLITE_OPEN_TRANSIENT_DB || eType==SQLITE_OPEN_WAL + ); assert( id!=0 ); UNUSED_PARAMETER(pVfs); pFile->h = INVALID_HANDLE_VALUE; @@ -31059,11 +31516,12 @@ /* If the second argument to this function is NULL, generate a ** temporary file name to use */ if( !zUtf8Name ){ - int rc = getTempname(MAX_PATH+1, zTmpname); + assert(isDelete && !isOpenJournal); + rc = getTempname(MAX_PATH+1, zTmpname); if( rc!=SQLITE_OK ){ return rc; } zUtf8Name = zTmpname; } @@ -31072,33 +31530,35 @@ zConverted = convertUtf8Filename(zUtf8Name); if( zConverted==0 ){ return SQLITE_NOMEM; } - if( flags & SQLITE_OPEN_READWRITE ){ + if( isReadWrite ){ dwDesiredAccess = GENERIC_READ | GENERIC_WRITE; }else{ dwDesiredAccess = GENERIC_READ; } + /* SQLITE_OPEN_EXCLUSIVE is used to make sure that a new file is ** created. SQLite doesn't use it to indicate "exclusive access" ** as it is usually understood. */ - assert(!(flags & SQLITE_OPEN_EXCLUSIVE) || (flags & SQLITE_OPEN_CREATE)); - if( flags & SQLITE_OPEN_EXCLUSIVE ){ + if( isExclusive ){ /* Creates a new file, only if it does not already exist. */ /* If the file exists, it fails. */ dwCreationDisposition = CREATE_NEW; - }else if( flags & SQLITE_OPEN_CREATE ){ + }else if( isCreate ){ /* Open existing file, or create if it doesn't exist */ dwCreationDisposition = OPEN_ALWAYS; }else{ /* Opens a file, only if it exists. */ dwCreationDisposition = OPEN_EXISTING; } + dwShareMode = FILE_SHARE_READ | FILE_SHARE_WRITE; - if( flags & SQLITE_OPEN_DELETEONCLOSE ){ + + if( isDelete ){ #if SQLITE_OS_WINCE dwFlagsAndAttributes = FILE_ATTRIBUTE_HIDDEN; isTemp = 1; #else dwFlagsAndAttributes = FILE_ATTRIBUTE_TEMPORARY @@ -31111,10 +31571,11 @@ /* Reports from the internet are that performance is always ** better if FILE_FLAG_RANDOM_ACCESS is used. Ticket #2699. */ #if SQLITE_OS_WINCE dwFlagsAndAttributes |= FILE_FLAG_RANDOM_ACCESS; #endif + if( isNT() ){ h = CreateFileW((WCHAR*)zConverted, dwDesiredAccess, dwShareMode, NULL, @@ -31136,41 +31597,45 @@ dwFlagsAndAttributes, NULL ); #endif } + OSTRACE(("OPEN %d %s 0x%lx %s\n", h, zName, dwDesiredAccess, h==INVALID_HANDLE_VALUE ? "failed" : "ok")); + if( h==INVALID_HANDLE_VALUE ){ pFile->lastErrno = GetLastError(); free(zConverted); - if( flags & SQLITE_OPEN_READWRITE ){ + if( isReadWrite ){ return winOpen(pVfs, zName, id, - ((flags|SQLITE_OPEN_READONLY)&~SQLITE_OPEN_READWRITE), pOutFlags); + ((flags|SQLITE_OPEN_READONLY)&~(SQLITE_OPEN_CREATE|SQLITE_OPEN_READWRITE)), pOutFlags); }else{ return SQLITE_CANTOPEN_BKPT; } } + if( pOutFlags ){ - if( flags & SQLITE_OPEN_READWRITE ){ + if( isReadWrite ){ *pOutFlags = SQLITE_OPEN_READWRITE; }else{ *pOutFlags = SQLITE_OPEN_READONLY; } } + memset(pFile, 0, sizeof(*pFile)); pFile->pMethod = &winIoMethod; pFile->h = h; pFile->lastErrno = NO_ERROR; pFile->pVfs = pVfs; pFile->pShm = 0; pFile->zPath = zName; pFile->sectorSize = getSectorSize(pVfs, zUtf8Name); + #if SQLITE_OS_WINCE - if( (flags & (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB)) == - (SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_DB) + if( isReadWrite && eType==SQLITE_OPEN_MAIN_DB && !winceCreateLock(zName, pFile) ){ CloseHandle(h); free(zConverted); return SQLITE_CANTOPEN_BKPT; @@ -31180,12 +31645,13 @@ }else #endif { free(zConverted); } + OpenCounter(+1); - return SQLITE_OK; + return rc; } /* ** Delete the named file. ** @@ -31699,10 +32165,17 @@ winSleep, /* xSleep */ winCurrentTime, /* xCurrentTime */ winGetLastError, /* xGetLastError */ winCurrentTimeInt64, /* xCurrentTimeInt64 */ }; + +#ifndef SQLITE_OMIT_WAL + /* get memory map allocation granularity */ + memset(&winSysInfo, 0, sizeof(SYSTEM_INFO)); + GetSystemInfo(&winSysInfo); + assert(winSysInfo.dwAllocationGranularity > 0); +#endif sqlite3_vfs_register(&winVfs, 1); return SQLITE_OK; } SQLITE_API int sqlite3_os_end(void){ @@ -31749,11 +32222,11 @@ ** start of a transaction, and is thus usually less than a few thousand, ** but can be as large as 2 billion for a really big database. */ /* Size of the Bitvec structure in bytes. */ -#define BITVEC_SZ (sizeof(void*)*128) /* 512 on 32bit. 1024 on 64bit */ +#define BITVEC_SZ 512 /* Round the union size down to the nearest pointer boundary, since that's how ** it will be aligned within the Bitvec struct. */ #define BITVEC_USIZE (((BITVEC_SZ-(3*sizeof(u32)))/sizeof(Bitvec*))*sizeof(Bitvec*)) @@ -32264,16 +32737,20 @@ ** Initialize and shutdown the page cache subsystem. Neither of these ** functions are threadsafe. */ SQLITE_PRIVATE int sqlite3PcacheInitialize(void){ if( sqlite3GlobalConfig.pcache.xInit==0 ){ + /* IMPLEMENTATION-OF: R-26801-64137 If the xInit() method is NULL, then the + ** built-in default page cache is used instead of the application defined + ** page cache. */ sqlite3PCacheSetDefault(); } return sqlite3GlobalConfig.pcache.xInit(sqlite3GlobalConfig.pcache.pArg); } SQLITE_PRIVATE void sqlite3PcacheShutdown(void){ if( sqlite3GlobalConfig.pcache.xShutdown ){ + /* IMPLEMENTATION-OF: R-26000-56589 The xShutdown() method may be NULL. */ sqlite3GlobalConfig.pcache.xShutdown(sqlite3GlobalConfig.pcache.pArg); } } /* @@ -32730,12 +33207,17 @@ typedef struct PCache1 PCache1; typedef struct PgHdr1 PgHdr1; typedef struct PgFreeslot PgFreeslot; -/* Pointers to structures of this type are cast and returned as -** opaque sqlite3_pcache* handles +/* Each page cache is an instance of the following object. Every +** open database file (including each in-memory database and each +** temporary or transient database) has a single page cache which +** is an instance of this object. +** +** Pointers to structures of this type are cast and returned as +** opaque sqlite3_pcache* handles. */ struct PCache1 { /* Cache configuration parameters. Page size (szPage) and the purgeable ** flag (bPurgeable) are set when the cache is created. nMax may be ** modified at any time by a call to the pcache1CacheSize() method. @@ -32791,10 +33273,13 @@ int nCurrentPage; /* Number of purgeable pages allocated */ PgHdr1 *pLruHead, *pLruTail; /* LRU list of unpinned pages */ /* Variables related to SQLITE_CONFIG_PAGECACHE settings. */ int szSlot; /* Size of each free slot */ + int nSlot; /* The number of pcache slots */ + int nFreeSlot; /* Number of unused pcache slots */ + int nReserve; /* Try to keep nFreeSlot above this */ void *pStart, *pEnd; /* Bounds of pagecache malloc range */ PgFreeslot *pFree; /* Free page blocks */ int isInit; /* True if initialized */ } pcache1_g; @@ -32838,10 +33323,12 @@ SQLITE_PRIVATE void sqlite3PCacheBufferSetup(void *pBuf, int sz, int n){ if( pcache1.isInit ){ PgFreeslot *p; sz = ROUNDDOWN8(sz); pcache1.szSlot = sz; + pcache1.nSlot = pcache1.nFreeSlot = n; + pcache1.nReserve = n>90 ? 10 : (n/10 + 1); pcache1.pStart = pBuf; pcache1.pFree = 0; while( n-- ){ p = (PgFreeslot*)pBuf; p->pNext = pcache1.pFree; @@ -32864,10 +33351,12 @@ sqlite3StatusSet(SQLITE_STATUS_PAGECACHE_SIZE, nByte); if( nByte<=pcache1.szSlot && pcache1.pFree ){ assert( pcache1.isInit ); p = (PgHdr1 *)pcache1.pFree; pcache1.pFree = pcache1.pFree->pNext; + pcache1.nFreeSlot--; + assert( pcache1.nFreeSlot>=0 ); sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, 1); }else{ /* Allocate a new buffer using sqlite3Malloc. Before doing so, exit the ** global pcache mutex and unlock the pager-cache object pCache. This is @@ -32897,10 +33386,12 @@ PgFreeslot *pSlot; sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_USED, -1); pSlot = (PgFreeslot*)p; pSlot->pNext = pcache1.pFree; pcache1.pFree = pSlot; + pcache1.nFreeSlot++; + assert( pcache1.nFreeSlot<=pcache1.nSlot ); }else{ int iSize; assert( sqlite3MemdebugHasType(p, MEMTYPE_PCACHE) ); sqlite3MemdebugSetType(p, MEMTYPE_HEAP); iSize = sqlite3MallocSize(p); @@ -32907,10 +33398,29 @@ sqlite3StatusAdd(SQLITE_STATUS_PAGECACHE_OVERFLOW, -iSize); sqlite3_free(p); } } +#ifdef SQLITE_ENABLE_MEMORY_MANAGEMENT +/* +** Return the size of a pcache allocation +*/ +static int pcache1MemSize(void *p){ + assert( sqlite3_mutex_held(pcache1.mutex) ); + if( p>=pcache1.pStart && p<pcache1.pEnd ){ + return pcache1.szSlot; + }else{ + int iSize; + assert( sqlite3MemdebugHasType(p, MEMTYPE_PCACHE) ); + sqlite3MemdebugSetType(p, MEMTYPE_HEAP); + iSize = sqlite3MallocSize(p); + sqlite3MemdebugSetType(p, MEMTYPE_PCACHE); + return iSize; + } +} +#endif /* SQLITE_ENABLE_MEMORY_MANAGEMENT */ + /* ** Allocate a new page object initially associated with cache pCache. */ static PgHdr1 *pcache1AllocPage(PCache1 *pCache){ int nByte = sizeof(PgHdr1) + pCache->szPage; @@ -32963,10 +33473,36 @@ pcache1EnterMutex(); pcache1Free(p); pcache1LeaveMutex(); } + +/* +** Return true if it desirable to avoid allocating a new page cache +** entry. +** +** If memory was allocated specifically to the page cache using +** SQLITE_CONFIG_PAGECACHE but that memory has all been used, then +** it is desirable to avoid allocating a new page cache entry because +** presumably SQLITE_CONFIG_PAGECACHE was suppose to be sufficient +** for all page cache needs and we should not need to spill the +** allocation onto the heap. +** +** Or, the heap is used for all page cache memory put the heap is +** under memory pressure, then again it is desirable to avoid +** allocating a new page cache entry in order to avoid stressing +** the heap even further. +*/ +static int pcache1UnderMemoryPressure(PCache1 *pCache){ + assert( sqlite3_mutex_held(pcache1.mutex) ); + if( pcache1.nSlot && pCache->szPage<=pcache1.szSlot ){ + return pcache1.nFreeSlot<pcache1.nReserve; + }else{ + return sqlite3HeapNearlyFull(); + } +} + /******************************************************************************/ /******** General Implementation Functions ************************************/ /* ** This function is used to resize the hash table used by the cache passed @@ -33204,18 +33740,20 @@ ** copy of the requested page. If one is found, it is returned. ** ** 2. If createFlag==0 and the page is not already in the cache, NULL is ** returned. ** -** 3. If createFlag is 1, and the page is not already in the cache, -** and if either of the following are true, return NULL: +** 3. If createFlag is 1, and the page is not already in the cache, then +** return NULL (do not allocate a new page) if any of the following +** conditions are true: ** ** (a) the number of pages pinned by the cache is greater than ** PCache1.nMax, or +** ** (b) the number of pages pinned by the cache is greater than ** the sum of nMax for all purgeable caches, less the sum of -** nMin for all other purgeable caches. +** nMin for all other purgeable caches, or ** ** 4. If none of the first three conditions apply and the cache is marked ** as purgeable, and if one of the following is true: ** ** (a) The number of pages allocated for the cache is already @@ -33222,10 +33760,13 @@ ** PCache1.nMax, or ** ** (b) The number of pages allocated for all purgeable caches is ** already equal to or greater than the sum of nMax for all ** purgeable caches, +** +** (c) The system is under memory pressure and wants to avoid +** unnecessary pages cache entry allocations ** ** then attempt to recycle a page from the LRU list. If it is the right ** size, return the recycled buffer. Otherwise, free the buffer and ** proceed to step 5. ** @@ -33254,10 +33795,11 @@ /* Step 3 of header comment. */ nPinned = pCache->nPage - pCache->nRecyclable; if( createFlag==1 && ( nPinned>=(pcache1.nMaxPage+pCache->nMin-pcache1.nMinPage) || nPinned>=(pCache->nMax * 9 / 10) + || pcache1UnderMemoryPressure(pCache) )){ goto fetch_out; } if( pCache->nPage>=pCache->nHash && pcache1ResizeHash(pCache) ){ @@ -33264,11 +33806,13 @@ goto fetch_out; } /* Step 4. Try to recycle a page buffer if appropriate. */ if( pCache->bPurgeable && pcache1.pLruTail && ( - (pCache->nPage+1>=pCache->nMax) || pcache1.nCurrentPage>=pcache1.nMaxPage + (pCache->nPage+1>=pCache->nMax) + || pcache1.nCurrentPage>=pcache1.nMaxPage + || pcache1UnderMemoryPressure(pCache) )){ pPage = pcache1.pLruTail; pcache1RemoveFromHash(pPage); pcache1PinPage(pPage); if( pPage->pCache->szPage!=pCache->szPage ){ @@ -33407,10 +33951,11 @@ ** ** Destroy a cache allocated using pcache1Create(). */ static void pcache1Destroy(sqlite3_pcache *p){ PCache1 *pCache = (PCache1 *)p; + assert( pCache->bPurgeable || (pCache->nMax==0 && pCache->nMin==0) ); pcache1EnterMutex(); pcache1TruncateUnsafe(pCache, 0); pcache1.nMaxPage -= pCache->nMax; pcache1.nMinPage -= pCache->nMin; pcache1EnforceMaxPage(); @@ -33454,12 +33999,12 @@ SQLITE_PRIVATE int sqlite3PcacheReleaseMemory(int nReq){ int nFree = 0; if( pcache1.pStart==0 ){ PgHdr1 *p; pcache1EnterMutex(); - while( (nReq<0 || nFree<nReq) && (p=pcache1.pLruTail) ){ - nFree += sqlite3MallocSize(PGHDR1_TO_PAGE(p)); + while( (nReq<0 || nFree<nReq) && ((p=pcache1.pLruTail)!=0) ){ + nFree += pcache1MemSize(PGHDR1_TO_PAGE(p)); pcache1PinPage(p); pcache1RemoveFromHash(p); pcache1FreePage(p); } pcache1LeaveMutex(); @@ -33964,11 +34509,11 @@ # define sqlite3WalOpen(x,y,z) 0 # define sqlite3WalClose(w,x,y,z) 0 # define sqlite3WalBeginReadTransaction(y,z) 0 # define sqlite3WalEndReadTransaction(z) # define sqlite3WalRead(v,w,x,y,z) 0 -# define sqlite3WalDbsize(y,z) +# define sqlite3WalDbsize(y) 0 # define sqlite3WalBeginWriteTransaction(y) 0 # define sqlite3WalEndWriteTransaction(x) 0 # define sqlite3WalUndo(x,y,z) 0 # define sqlite3WalSavepoint(y,z) # define sqlite3WalSavepointUndo(y,z) 0 @@ -34000,13 +34545,12 @@ SQLITE_PRIVATE void sqlite3WalEndReadTransaction(Wal *pWal); /* Read a page from the write-ahead log, if it is present. */ SQLITE_PRIVATE int sqlite3WalRead(Wal *pWal, Pgno pgno, int *pInWal, int nOut, u8 *pOut); -/* Return the size of the database as it existed at the beginning -** of the snapshot */ -SQLITE_PRIVATE void sqlite3WalDbsize(Wal *pWal, Pgno *pPgno); +/* If the WAL is not empty, return the size of the database. */ +SQLITE_PRIVATE Pgno sqlite3WalDbsize(Wal *pWal); /* Obtain or release the WRITER lock. */ SQLITE_PRIVATE int sqlite3WalBeginWriteTransaction(Wal *pWal); SQLITE_PRIVATE int sqlite3WalEndWriteTransaction(Wal *pWal); @@ -34048,12 +34592,16 @@ #endif /* _WAL_H_ */ /************** End of wal.h *************************************************/ /************** Continuing where we left off in pager.c **********************/ -/* -******************** NOTES ON THE DESIGN OF THE PAGER ************************ + +/******************* NOTES ON THE DESIGN OF THE PAGER ************************ +** +** This comment block describes invariants that hold when using a rollback +** journal. These invariants do not apply for journal_mode=WAL, +** journal_mode=MEMORY, or journal_mode=OFF. ** ** Within this comment block, a page is deemed to have been synced ** automatically as soon as it is written when PRAGMA synchronous=OFF. ** Otherwise, the page is not synced until the xSync method of the VFS ** is called successfully on the file containing the page. @@ -34083,11 +34631,11 @@ ** both the content in the database when the rollback journal was written ** and the content in the database at the beginning of the current ** transaction. ** ** (3) Writes to the database file are an integer multiple of the page size -** in length and are aligned to a page boundary. +** in length and are aligned on a page boundary. ** ** (4) Reads from the database file are either aligned on a page boundary and ** an integer multiple of the page size in length or are taken from the ** first 100 bytes of the database file. ** @@ -34114,11 +34662,12 @@ ** method is a no-op, but that does not change the fact the SQLite will ** invoke it.) ** ** (9) Whenever the database file is modified, at least one bit in the range ** of bytes from 24 through 39 inclusive will be changed prior to releasing -** the EXCLUSIVE lock. +** the EXCLUSIVE lock, thus signaling other connections on the same +** database to flush their caches. ** ** (10) The pattern of bits in bytes 24 through 39 shall not repeat in less ** than one billion transactions. ** ** (11) A database file is well-formed at the beginning and at the conclusion @@ -34127,11 +34676,12 @@ ** (12) An EXCLUSIVE lock is held on the database file when writing to ** the database file. ** ** (13) A SHARED lock is held on the database file while reading any ** content out of the database file. -*/ +** +******************************************************************************/ /* ** Macros for troubleshooting. Normally turned off */ #if 0 @@ -34152,62 +34702,283 @@ */ #define PAGERID(p) ((int)(p->fd)) #define FILEHANDLEID(fd) ((int)fd) /* -** The page cache as a whole is always in one of the following -** states: -** -** PAGER_UNLOCK The page cache is not currently reading or -** writing the database file. There is no -** data held in memory. This is the initial -** state. -** -** PAGER_SHARED The page cache is reading the database. -** Writing is not permitted. There can be -** multiple readers accessing the same database -** file at the same time. -** -** PAGER_RESERVED This process has reserved the database for writing -** but has not yet made any changes. Only one process -** at a time can reserve the database. The original -** database file has not been modified so other -** processes may still be reading the on-disk -** database file. -** -** PAGER_EXCLUSIVE The page cache is writing the database. -** Access is exclusive. No other processes or -** threads can be reading or writing while one -** process is writing. -** -** PAGER_SYNCED The pager moves to this state from PAGER_EXCLUSIVE -** after all dirty pages have been written to the -** database file and the file has been synced to -** disk. All that remains to do is to remove or -** truncate the journal file and the transaction -** will be committed. -** -** The page cache comes up in PAGER_UNLOCK. The first time a -** sqlite3PagerGet() occurs, the state transitions to PAGER_SHARED. -** After all pages have been released using sqlite_page_unref(), -** the state transitions back to PAGER_UNLOCK. The first time -** that sqlite3PagerWrite() is called, the state transitions to -** PAGER_RESERVED. (Note that sqlite3PagerWrite() can only be -** called on an outstanding page which means that the pager must -** be in PAGER_SHARED before it transitions to PAGER_RESERVED.) -** PAGER_RESERVED means that there is an open rollback journal. -** The transition to PAGER_EXCLUSIVE occurs before any changes -** are made to the database file, though writes to the rollback -** journal occurs with just PAGER_RESERVED. After an sqlite3PagerRollback() -** or sqlite3PagerCommitPhaseTwo(), the state can go back to PAGER_SHARED, -** or it can stay at PAGER_EXCLUSIVE if we are in exclusive access mode. -*/ -#define PAGER_UNLOCK 0 -#define PAGER_SHARED 1 /* same as SHARED_LOCK */ -#define PAGER_RESERVED 2 /* same as RESERVED_LOCK */ -#define PAGER_EXCLUSIVE 4 /* same as EXCLUSIVE_LOCK */ -#define PAGER_SYNCED 5 +** The Pager.eState variable stores the current 'state' of a pager. A +** pager may be in any one of the seven states shown in the following +** state diagram. +** +** OPEN <------+------+ +** | | | +** V | | +** +---------> READER-------+ | +** | | | +** | V | +** |<-------WRITER_LOCKED------> ERROR +** | | ^ +** | V | +** |<------WRITER_CACHEMOD-------->| +** | | | +** | V | +** |<-------WRITER_DBMOD---------->| +** | | | +** | V | +** +<------WRITER_FINISHED-------->+ +** +** +** List of state transitions and the C [function] that performs each: +** +** OPEN -> READER [sqlite3PagerSharedLock] +** READER -> OPEN [pager_unlock] +** +** READER -> WRITER_LOCKED [sqlite3PagerBegin] +** WRITER_LOCKED -> WRITER_CACHEMOD [pager_open_journal] +** WRITER_CACHEMOD -> WRITER_DBMOD [syncJournal] +** WRITER_DBMOD -> WRITER_FINISHED [sqlite3PagerCommitPhaseOne] +** WRITER_*** -> READER [pager_end_transaction] +** +** WRITER_*** -> ERROR [pager_error] +** ERROR -> OPEN [pager_unlock] +** +** +** OPEN: +** +** The pager starts up in this state. Nothing is guaranteed in this +** state - the file may or may not be locked and the database size is +** unknown. The database may not be read or written. +** +** * No read or write transaction is active. +** * Any lock, or no lock at all, may be held on the database file. +** * The dbSize, dbOrigSize and dbFileSize variables may not be trusted. +** +** READER: +** +** In this state all the requirements for reading the database in +** rollback (non-WAL) mode are met. Unless the pager is (or recently +** was) in exclusive-locking mode, a user-level read transaction is +** open. The database size is known in this state. +** +** A connection running with locking_mode=normal enters this state when +** it opens a read-transaction on the database and returns to state +** OPEN after the read-transaction is completed. However a connection +** running in locking_mode=exclusive (including temp databases) remains in +** this state even after the read-transaction is closed. The only way +** a locking_mode=exclusive connection can transition from READER to OPEN +** is via the ERROR state (see below). +** +** * A read transaction may be active (but a write-transaction cannot). +** * A SHARED or greater lock is held on the database file. +** * The dbSize variable may be trusted (even if a user-level read +** transaction is not active). The dbOrigSize and dbFileSize variables +** may not be trusted at this point. +** * If the database is a WAL database, then the WAL connection is open. +** * Even if a read-transaction is not open, it is guaranteed that +** there is no hot-journal in the file-system. +** +** WRITER_LOCKED: +** +** The pager moves to this state from READER when a write-transaction +** is first opened on the database. In WRITER_LOCKED state, all locks +** required to start a write-transaction are held, but no actual +** modifications to the cache or database have taken place. +** +** In rollback mode, a RESERVED or (if the transaction was opened with +** BEGIN EXCLUSIVE) EXCLUSIVE lock is obtained on the database file when +** moving to this state, but the journal file is not written to or opened +** to in this state. If the transaction is committed or rolled back while +** in WRITER_LOCKED state, all that is required is to unlock the database +** file. +** +** IN WAL mode, WalBeginWriteTransaction() is called to lock the log file. +** If the connection is running with locking_mode=exclusive, an attempt +** is made to obtain an EXCLUSIVE lock on the database file. +** +** * A write transaction is active. +** * If the connection is open in rollback-mode, a RESERVED or greater +** lock is held on the database file. +** * If the connection is open in WAL-mode, a WAL write transaction +** is open (i.e. sqlite3WalBeginWriteTransaction() has been successfully +** called). +** * The dbSize, dbOrigSize and dbFileSize variables are all valid. +** * The contents of the pager cache have not been modified. +** * The journal file may or may not be open. +** * Nothing (not even the first header) has been written to the journal. +** +** WRITER_CACHEMOD: +** +** A pager moves from WRITER_LOCKED state to this state when a page is +** first modified by the upper layer. In rollback mode the journal file +** is opened (if it is not already open) and a header written to the +** start of it. The database file on disk has not been modified. +** +** * A write transaction is active. +** * A RESERVED or greater lock is held on the database file. +** * The journal file is open and the first header has been written +** to it, but the header has not been synced to disk. +** * The contents of the page cache have been modified. +** +** WRITER_DBMOD: +** +** The pager transitions from WRITER_CACHEMOD into WRITER_DBMOD state +** when it modifies the contents of the database file. WAL connections +** never enter this state (since they do not modify the database file, +** just the log file). +** +** * A write transaction is active. +** * An EXCLUSIVE or greater lock is held on the database file. +** * The journal file is open and the first header has been written +** and synced to disk. +** * The contents of the page cache have been modified (and possibly +** written to disk). +** +** WRITER_FINISHED: +** +** It is not possible for a WAL connection to enter this state. +** +** A rollback-mode pager changes to WRITER_FINISHED state from WRITER_DBMOD +** state after the entire transaction has been successfully written into the +** database file. In this state the transaction may be committed simply +** by finalizing the journal file. Once in WRITER_FINISHED state, it is +** not possible to modify the database further. At this point, the upper +** layer must either commit or rollback the transaction. +** +** * A write transaction is active. +** * An EXCLUSIVE or greater lock is held on the database file. +** * All writing and syncing of journal and database data has finished. +** If no error occured, all that remains is to finalize the journal to +** commit the transaction. If an error did occur, the caller will need +** to rollback the transaction. +** +** ERROR: +** +** The ERROR state is entered when an IO or disk-full error (including +** SQLITE_IOERR_NOMEM) occurs at a point in the code that makes it +** difficult to be sure that the in-memory pager state (cache contents, +** db size etc.) are consistent with the contents of the file-system. +** +** Temporary pager files may enter the ERROR state, but in-memory pagers +** cannot. +** +** For example, if an IO error occurs while performing a rollback, +** the contents of the page-cache may be left in an inconsistent state. +** At this point it would be dangerous to change back to READER state +** (as usually happens after a rollback). Any subsequent readers might +** report database corruption (due to the inconsistent cache), and if +** they upgrade to writers, they may inadvertently corrupt the database +** file. To avoid this hazard, the pager switches into the ERROR state +** instead of READER following such an error. +** +** Once it has entered the ERROR state, any attempt to use the pager +** to read or write data returns an error. Eventually, once all +** outstanding transactions have been abandoned, the pager is able to +** transition back to OPEN state, discarding the contents of the +** page-cache and any other in-memory state at the same time. Everything +** is reloaded from disk (and, if necessary, hot-journal rollback peformed) +** when a read-transaction is next opened on the pager (transitioning +** the pager into READER state). At that point the system has recovered +** from the error. +** +** Specifically, the pager jumps into the ERROR state if: +** +** 1. An error occurs while attempting a rollback. This happens in +** function sqlite3PagerRollback(). +** +** 2. An error occurs while attempting to finalize a journal file +** following a commit in function sqlite3PagerCommitPhaseTwo(). +** +** 3. An error occurs while attempting to write to the journal or +** database file in function pagerStress() in order to free up +** memory. +** +** In other cases, the error is returned to the b-tree layer. The b-tree +** layer then attempts a rollback operation. If the error condition +** persists, the pager enters the ERROR state via condition (1) above. +** +** Condition (3) is necessary because it can be triggered by a read-only +** statement executed within a transaction. In this case, if the error +** code were simply returned to the user, the b-tree layer would not +** automatically attempt a rollback, as it assumes that an error in a +** read-only statement cannot leave the pager in an internally inconsistent +** state. +** +** * The Pager.errCode variable is set to something other than SQLITE_OK. +** * There are one or more outstanding references to pages (after the +** last reference is dropped the pager should move back to OPEN state). +** * The pager is not an in-memory pager. +** +** +** Notes: +** +** * A pager is never in WRITER_DBMOD or WRITER_FINISHED state if the +** connection is open in WAL mode. A WAL connection is always in one +** of the first four states. +** +** * Normally, a connection open in exclusive mode is never in PAGER_OPEN +** state. There are two exceptions: immediately after exclusive-mode has +** been turned on (and before any read or write transactions are +** executed), and when the pager is leaving the "error state". +** +** * See also: assert_pager_state(). +*/ +#define PAGER_OPEN 0 +#define PAGER_READER 1 +#define PAGER_WRITER_LOCKED 2 +#define PAGER_WRITER_CACHEMOD 3 +#define PAGER_WRITER_DBMOD 4 +#define PAGER_WRITER_FINISHED 5 +#define PAGER_ERROR 6 + +/* +** The Pager.eLock variable is almost always set to one of the +** following locking-states, according to the lock currently held on +** the database file: NO_LOCK, SHARED_LOCK, RESERVED_LOCK or EXCLUSIVE_LOCK. +** This variable is kept up to date as locks are taken and released by +** the pagerLockDb() and pagerUnlockDb() wrappers. +** +** If the VFS xLock() or xUnlock() returns an error other than SQLITE_BUSY +** (i.e. one of the SQLITE_IOERR subtypes), it is not clear whether or not +** the operation was successful. In these circumstances pagerLockDb() and +** pagerUnlockDb() take a conservative approach - eLock is always updated +** when unlocking the file, and only updated when locking the file if the +** VFS call is successful. This way, the Pager.eLock variable may be set +** to a less exclusive (lower) value than the lock that is actually held +** at the system level, but it is never set to a more exclusive value. +** +** This is usually safe. If an xUnlock fails or appears to fail, there may +** be a few redundant xLock() calls or a lock may be held for longer than +** required, but nothing really goes wrong. +** +** The exception is when the database file is unlocked as the pager moves +** from ERROR to OPEN state. At this point there may be a hot-journal file +** in the file-system that needs to be rolled back (as part of a OPEN->SHARED +** transition, by the same pager or any other). If the call to xUnlock() +** fails at this point and the pager is left holding an EXCLUSIVE lock, this +** can confuse the call to xCheckReservedLock() call made later as part +** of hot-journal detection. +** +** xCheckReservedLock() is defined as returning true "if there is a RESERVED +** lock held by this process or any others". So xCheckReservedLock may +** return true because the caller itself is holding an EXCLUSIVE lock (but +** doesn't know it because of a previous error in xUnlock). If this happens +** a hot-journal may be mistaken for a journal being created by an active +** transaction in another process, causing SQLite to read from the database +** without rolling it back. +** +** To work around this, if a call to xUnlock() fails when unlocking the +** database in the ERROR state, Pager.eLock is set to UNKNOWN_LOCK. It +** is only changed back to a real locking state after a successful call +** to xLock(EXCLUSIVE). Also, the code to do the OPEN->SHARED state transition +** omits the check for a hot-journal if Pager.eLock is set to UNKNOWN_LOCK +** lock. Instead, it assumes a hot-journal exists and obtains an EXCLUSIVE +** lock on the database file before attempting to roll it back. See function +** PagerSharedLock() for more detail. +** +** Pager.eLock may only be set to UNKNOWN_LOCK when the pager is in +** PAGER_OPEN state. +*/ +#define UNKNOWN_LOCK (EXCLUSIVE_LOCK+1) /* ** A macro used for invoking the codec if there is one */ #ifdef SQLITE_HAS_CODEC @@ -34253,37 +35024,32 @@ u32 aWalData[WAL_SAVEPOINT_NDATA]; /* WAL savepoint context */ #endif }; /* -** A open page cache is an instance of the following structure. -** -** errCode -** -** Pager.errCode may be set to SQLITE_IOERR, SQLITE_CORRUPT, or -** or SQLITE_FULL. Once one of the first three errors occurs, it persists -** and is returned as the result of every major pager API call. The -** SQLITE_FULL return code is slightly different. It persists only until the -** next successful rollback is performed on the pager cache. Also, -** SQLITE_FULL does not affect the sqlite3PagerGet() and sqlite3PagerLookup() -** APIs, they may still be used successfully. -** -** dbSizeValid, dbSize, dbOrigSize, dbFileSize -** -** Managing the size of the database file in pages is a little complicated. -** The variable Pager.dbSize contains the number of pages that the database -** image currently contains. As the database image grows or shrinks this -** variable is updated. The variable Pager.dbFileSize contains the number -** of pages in the database file. This may be different from Pager.dbSize -** if some pages have been appended to the database image but not yet written -** out from the cache to the actual file on disk. Or if the image has been -** truncated by an incremental-vacuum operation. The Pager.dbOrigSize variable -** contains the number of pages in the database image when the current -** transaction was opened. The contents of all three of these variables is -** only guaranteed to be correct if the boolean Pager.dbSizeValid is true. -** -** TODO: Under what conditions is dbSizeValid set? Cleared? +** A open page cache is an instance of struct Pager. A description of +** some of the more important member variables follows: +** +** eState +** +** The current 'state' of the pager object. See the comment and state +** diagram above for a description of the pager state. +** +** eLock +** +** For a real on-disk database, the current lock held on the database file - +** NO_LOCK, SHARED_LOCK, RESERVED_LOCK or EXCLUSIVE_LOCK. +** +** For a temporary or in-memory database (neither of which require any +** locks), this variable is always set to EXCLUSIVE_LOCK. Since such +** databases always have Pager.exclusiveMode==1, this tricks the pager +** logic into thinking that it already has all the locks it will ever +** need (and no reason to release them). +** +** In some (obscure) circumstances, this variable may also be set to +** UNKNOWN_LOCK. See the comment above the #define of UNKNOWN_LOCK for +** details. ** ** changeCountDone ** ** This boolean variable is used to make sure that the change-counter ** (the 4-byte header field at byte offset 24 of the database file) is @@ -34298,28 +35064,10 @@ ** ** This mechanism means that when running in exclusive mode, a connection ** need only update the change-counter once, for the first transaction ** committed. ** -** dbModified -** -** The dbModified flag is set whenever a database page is dirtied. -** It is cleared at the end of each transaction. -** -** It is used when committing or otherwise ending a transaction. If -** the dbModified flag is clear then less work has to be done. -** -** journalStarted -** -** This flag is set during a write-transaction after the first -** journal-header is written and synced to disk. -** -** After this has happened, new pages appended to the database -** do not need the PGHDR_NEED_SYNC flag set, as they do not need -** to wait for a journal sync before they can be written out to -** the database file (see function pager_write()). -** ** setMaster ** ** When PagerCommitPhaseOne() is called to commit a transaction, it may ** (or may not) specify a master-journal name to be written into the ** journal file before it is synced to disk. @@ -34326,84 +35074,146 @@ ** ** Whether or not a journal file contains a master-journal pointer affects ** the way in which the journal file is finalized after the transaction is ** committed or rolled back when running in "journal_mode=PERSIST" mode. ** If a journal file does not contain a master-journal pointer, it is -** finalized by overwriting the first journal header with zeroes. If, -** on the other hand, it does contain a master-journal pointer, the -** journal file is finalized by truncating it to zero bytes, just as if -** the connection were running in "journal_mode=truncate" mode. +** finalized by overwriting the first journal header with zeroes. If +** it does contain a master-journal pointer the journal file is finalized +** by truncating it to zero bytes, just as if the connection were +** running in "journal_mode=truncate" mode. ** ** Journal files that contain master journal pointers cannot be finalized ** simply by overwriting the first journal-header with zeroes, as the ** master journal pointer could interfere with hot-journal rollback of any ** subsequently interrupted transaction that reuses the journal file. ** ** The flag is cleared as soon as the journal file is finalized (either ** by PagerCommitPhaseTwo or PagerRollback). If an IO error prevents the ** journal file from being successfully finalized, the setMaster flag -** is cleared anyway. +** is cleared anyway (and the pager will move to ERROR state). ** ** doNotSpill, doNotSyncSpill ** -** When enabled, cache spills are prohibited. The doNotSpill variable -** inhibits all cache spill and doNotSyncSpill inhibits those spills that -** would require a journal sync. The doNotSyncSpill is set and cleared -** by sqlite3PagerWrite() in order to prevent a journal sync from happening -** in between the journalling of two pages on the same sector. The -** doNotSpill value set to prevent pagerStress() from trying to use -** the journal during a rollback. -** -** needSync -** -** TODO: It might be easier to set this variable in writeJournalHdr() -** and writeMasterJournal() only. Change its meaning to "unsynced data -** has been written to the journal". +** These two boolean variables control the behaviour of cache-spills +** (calls made by the pcache module to the pagerStress() routine to +** write cached data to the file-system in order to free up memory). +** +** When doNotSpill is non-zero, writing to the database from pagerStress() +** is disabled altogether. This is done in a very obscure case that +** comes up during savepoint rollback that requires the pcache module +** to allocate a new page to prevent the journal file from being written +** while it is being traversed by code in pager_playback(). +** +** If doNotSyncSpill is non-zero, writing to the database from pagerStress() +** is permitted, but syncing the journal file is not. This flag is set +** by sqlite3PagerWrite() when the file-system sector-size is larger than +** the database page-size in order to prevent a journal sync from happening +** in between the journalling of two pages on the same sector. ** ** subjInMemory ** ** This is a boolean variable. If true, then any required sub-journal ** is opened as an in-memory journal file. If false, then in-memory ** sub-journals are only used for in-memory pager files. +** +** This variable is updated by the upper layer each time a new +** write-transaction is opened. +** +** dbSize, dbOrigSize, dbFileSize +** +** Variable dbSize is set to the number of pages in the database file. +** It is valid in PAGER_READER and higher states (all states except for +** OPEN and ERROR). +** +** dbSize is set based on the size of the database file, which may be +** larger than the size of the database (the value stored at offset +** 28 of the database header by the btree). If the size of the file +** is not an integer multiple of the page-size, the value stored in +** dbSize is rounded down (i.e. a 5KB file with 2K page-size has dbSize==2). +** Except, any file that is greater than 0 bytes in size is considered +** to have at least one page. (i.e. a 1KB file with 2K page-size leads +** to dbSize==1). +** +** During a write-transaction, if pages with page-numbers greater than +** dbSize are modified in the cache, dbSize is updated accordingly. +** Similarly, if the database is truncated using PagerTruncateImage(), +** dbSize is updated. +** +** Variables dbOrigSize and dbFileSize are valid in states +** PAGER_WRITER_LOCKED and higher. dbOrigSize is a copy of the dbSize +** variable at the start of the transaction. It is used during rollback, +** and to determine whether or not pages need to be journalled before +** being modified. +** +** Throughout a write-transaction, dbFileSize contains the size of +** the file on disk in pages. It is set to a copy of dbSize when the +** write-transaction is first opened, and updated when VFS calls are made +** to write or truncate the database file on disk. +** +** The only reason the dbFileSize variable is required is to suppress +** unnecessary calls to xTruncate() after committing a transaction. If, +** when a transaction is committed, the dbFileSize variable indicates +** that the database file is larger than the database image (Pager.dbSize), +** pager_truncate() is called. The pager_truncate() call uses xFilesize() +** to measure the database file on disk, and then truncates it if required. +** dbFileSize is not used when rolling back a transaction. In this case +** pager_truncate() is called unconditionally (which means there may be +** a call to xFilesize() that is not strictly required). In either case, +** pager_truncate() may cause the file to become smaller or larger. +** +** dbHintSize +** +** The dbHintSize variable is used to limit the number of calls made to +** the VFS xFileControl(FCNTL_SIZE_HINT) method. +** +** dbHintSize is set to a copy of the dbSize variable when a +** write-transaction is opened (at the same time as dbFileSize and +** dbOrigSize). If the xFileControl(FCNTL_SIZE_HINT) method is called, +** dbHintSize is increased to the number of pages that correspond to the +** size-hint passed to the method call. See pager_write_pagelist() for +** details. +** +** errCode +** +** The Pager.errCode variable is only ever used in PAGER_ERROR state. It +** is set to zero in all other states. In PAGER_ERROR state, Pager.errCode +** is always set to SQLITE_FULL, SQLITE_IOERR or one of the SQLITE_IOERR_XXX +** sub-codes. */ struct Pager { sqlite3_vfs *pVfs; /* OS functions to use for IO */ u8 exclusiveMode; /* Boolean. True if locking_mode==EXCLUSIVE */ - u8 journalMode; /* On of the PAGER_JOURNALMODE_* values */ + u8 journalMode; /* One of the PAGER_JOURNALMODE_* values */ u8 useJournal; /* Use a rollback journal on this file */ u8 noReadlock; /* Do not bother to obtain readlocks */ u8 noSync; /* Do not sync the journal if true */ u8 fullSync; /* Do extra syncs of the journal for robustness */ u8 sync_flags; /* One of SYNC_NORMAL or SYNC_FULL */ u8 tempFile; /* zFilename is a temporary file */ u8 readOnly; /* True for a read-only database */ u8 memDb; /* True to inhibit all file I/O */ - /* The following block contains those class members that are dynamically - ** modified during normal operations. The other variables in this structure - ** are either constant throughout the lifetime of the pager, or else - ** used to store configuration parameters that affect the way the pager - ** operates. - ** - ** The 'state' variable is described in more detail along with the - ** descriptions of the values it may take - PAGER_UNLOCK etc. Many of the - ** other variables in this block are described in the comment directly - ** above this class definition. + /************************************************************************** + ** The following block contains those class members that change during + ** routine opertion. Class members not in this block are either fixed + ** when the pager is first created or else only change when there is a + ** significant mode change (such as changing the page_size, locking_mode, + ** or the journal_mode). From another view, these class members describe + ** the "state" of the pager, while other class members describe the + ** "configuration" of the pager. */ - u8 state; /* PAGER_UNLOCK, _SHARED, _RESERVED, etc. */ - u8 dbModified; /* True if there are any changes to the Db */ - u8 needSync; /* True if an fsync() is needed on the journal */ - u8 journalStarted; /* True if header of journal is synced */ + u8 eState; /* Pager state (OPEN, READER, WRITER_LOCKED..) */ + u8 eLock; /* Current lock held on database file */ u8 changeCountDone; /* Set after incrementing the change-counter */ u8 setMaster; /* True if a m-j name has been written to jrnl */ u8 doNotSpill; /* Do not spill the cache when non-zero */ u8 doNotSyncSpill; /* Do not do a spill that requires jrnl sync */ - u8 dbSizeValid; /* Set when dbSize is correct */ u8 subjInMemory; /* True to use in-memory sub-journals */ Pgno dbSize; /* Number of pages in the database */ Pgno dbOrigSize; /* dbSize before the current transaction */ Pgno dbFileSize; /* Number of pages in the database file */ + Pgno dbHintSize; /* Value passed to FCNTL_SIZE_HINT call */ int errCode; /* One of several kinds of errors */ int nRec; /* Pages journalled since last j-header written */ u32 cksumInit; /* Quasi-random value added to every checksum */ u32 nSubRec; /* Number of records written to sub-journal */ Bitvec *pInJournal; /* One bit for each page in the database file */ @@ -34410,21 +35220,25 @@ sqlite3_file *fd; /* File descriptor for database */ sqlite3_file *jfd; /* File descriptor for main journal */ sqlite3_file *sjfd; /* File descriptor for sub-journal */ i64 journalOff; /* Current write offset in the journal file */ i64 journalHdr; /* Byte offset to previous journal header */ - i64 journalSizeLimit; /* Size limit for persistent journal files */ + sqlite3_backup *pBackup; /* Pointer to list of ongoing backup processes */ PagerSavepoint *aSavepoint; /* Array of active savepoints */ int nSavepoint; /* Number of elements in aSavepoint[] */ char dbFileVers[16]; /* Changes whenever database file changes */ - u32 sectorSize; /* Assumed sector size during rollback */ + /* + ** End of the routinely-changing class members + ***************************************************************************/ u16 nExtra; /* Add this many bytes to each in-memory page */ i16 nReserve; /* Number of unused bytes at end of each page */ u32 vfsFlags; /* Flags for sqlite3_vfs.xOpen() */ + u32 sectorSize; /* Assumed sector size during rollback */ int pageSize; /* Number of bytes in a page */ Pgno mxPgno; /* Maximum allowed size of the database */ + i64 journalSizeLimit; /* Size limit for persistent journal files */ char *zFilename; /* Name of the database file */ char *zJournal; /* Name of the journal file */ int (*xBusyHandler)(void*); /* Function to call when busy */ void *pBusyHandlerArg; /* Context argument for xBusyHandler */ #ifdef SQLITE_TEST @@ -34438,11 +35252,10 @@ void (*xCodecFree)(void*); /* Destructor for the codec */ void *pCodec; /* First argument to xCodec... methods */ #endif char *pTmpSpace; /* Pager.pageSize bytes of space for tmp use */ PCache *pPCache; /* Pointer to page cache object */ - sqlite3_backup *pBackup; /* Pointer to list of ongoing backup processes */ #ifndef SQLITE_OMIT_WAL Wal *pWal; /* Write-ahead log used by "journal_mode=wal" */ char *zWal; /* File name for write-ahead log */ #endif }; @@ -34517,26 +35330,225 @@ /* ** The maximum legal page number is (2^31 - 1). */ #define PAGER_MAX_PGNO 2147483647 +/* +** The argument to this macro is a file descriptor (type sqlite3_file*). +** Return 0 if it is not open, or non-zero (but not 1) if it is. +** +** This is so that expressions can be written as: +** +** if( isOpen(pPager->jfd) ){ ... +** +** instead of +** +** if( pPager->jfd->pMethods ){ ... +*/ +#define isOpen(pFd) ((pFd)->pMethods) + +/* +** Return true if this pager uses a write-ahead log instead of the usual +** rollback journal. Otherwise false. +*/ +#ifndef SQLITE_OMIT_WAL +static int pagerUseWal(Pager *pPager){ + return (pPager->pWal!=0); +} +#else +# define pagerUseWal(x) 0 +# define pagerRollbackWal(x) 0 +# define pagerWalFrames(v,w,x,y,z) 0 +# define pagerOpenWalIfPresent(z) SQLITE_OK +# define pagerBeginReadTransaction(z) SQLITE_OK +#endif + #ifndef NDEBUG /* ** Usage: ** ** assert( assert_pager_state(pPager) ); +** +** This function runs many asserts to try to find inconsistencies in +** the internal state of the Pager object. */ -static int assert_pager_state(Pager *pPager){ +static int assert_pager_state(Pager *p){ + Pager *pPager = p; - /* A temp-file is always in PAGER_EXCLUSIVE or PAGER_SYNCED state. */ - assert( pPager->tempFile==0 || pPager->state>=PAGER_EXCLUSIVE ); + /* State must be valid. */ + assert( p->eState==PAGER_OPEN + || p->eState==PAGER_READER + || p->eState==PAGER_WRITER_LOCKED + || p->eState==PAGER_WRITER_CACHEMOD + || p->eState==PAGER_WRITER_DBMOD + || p->eState==PAGER_WRITER_FINISHED + || p->eState==PAGER_ERROR + ); - /* The changeCountDone flag is always set for temp-files */ - assert( pPager->tempFile==0 || pPager->changeCountDone ); + /* Regardless of the current state, a temp-file connection always behaves + ** as if it has an exclusive lock on the database file. It never updates + ** the change-counter field, so the changeCountDone flag is always set. + */ + assert( p->tempFile==0 || p->eLock==EXCLUSIVE_LOCK ); + assert( p->tempFile==0 || pPager->changeCountDone ); + + /* If the useJournal flag is clear, the journal-mode must be "OFF". + ** And if the journal-mode is "OFF", the journal file must not be open. + */ + assert( p->journalMode==PAGER_JOURNALMODE_OFF || p->useJournal ); + assert( p->journalMode!=PAGER_JOURNALMODE_OFF || !isOpen(p->jfd) ); + + /* Check that MEMDB implies noSync. And an in-memory journal. Since + ** this means an in-memory pager performs no IO at all, it cannot encounter + ** either SQLITE_IOERR or SQLITE_FULL during rollback or while finalizing + ** a journal file. (although the in-memory journal implementation may + ** return SQLITE_IOERR_NOMEM while the journal file is being written). It + ** is therefore not possible for an in-memory pager to enter the ERROR + ** state. + */ + if( MEMDB ){ + assert( p->noSync ); + assert( p->journalMode==PAGER_JOURNALMODE_OFF + || p->journalMode==PAGER_JOURNALMODE_MEMORY + ); + assert( p->eState!=PAGER_ERROR && p->eState!=PAGER_OPEN ); + assert( pagerUseWal(p)==0 ); + } + + /* If changeCountDone is set, a RESERVED lock or greater must be held + ** on the file. + */ + assert( pPager->changeCountDone==0 || pPager->eLock>=RESERVED_LOCK ); + assert( p->eLock!=PENDING_LOCK ); + + switch( p->eState ){ + case PAGER_OPEN: + assert( !MEMDB ); + assert( pPager->errCode==SQLITE_OK ); + assert( sqlite3PcacheRefCount(pPager->pPCache)==0 || pPager->tempFile ); + break; + + case PAGER_READER: + assert( pPager->errCode==SQLITE_OK ); + assert( p->eLock!=UNKNOWN_LOCK ); + assert( p->eLock>=SHARED_LOCK || p->noReadlock ); + break; + + case PAGER_WRITER_LOCKED: + assert( p->eLock!=UNKNOWN_LOCK ); + assert( pPager->errCode==SQLITE_OK ); + if( !pagerUseWal(pPager) ){ + assert( p->eLock>=RESERVED_LOCK ); + } + assert( pPager->dbSize==pPager->dbOrigSize ); + assert( pPager->dbOrigSize==pPager->dbFileSize ); + assert( pPager->dbOrigSize==pPager->dbHintSize ); + assert( pPager->setMaster==0 ); + break; + + case PAGER_WRITER_CACHEMOD: + assert( p->eLock!=UNKNOWN_LOCK ); + assert( pPager->errCode==SQLITE_OK ); + if( !pagerUseWal(pPager) ){ + /* It is possible that if journal_mode=wal here that neither the + ** journal file nor the WAL file are open. This happens during + ** a rollback transaction that switches from journal_mode=off + ** to journal_mode=wal. + */ + assert( p->eLock>=RESERVED_LOCK ); + assert( isOpen(p->jfd) + || p->journalMode==PAGER_JOURNALMODE_OFF + || p->journalMode==PAGER_JOURNALMODE_WAL + ); + } + assert( pPager->dbOrigSize==pPager->dbFileSize ); + assert( pPager->dbOrigSize==pPager->dbHintSize ); + break; + + case PAGER_WRITER_DBMOD: + assert( p->eLock==EXCLUSIVE_LOCK ); + assert( pPager->errCode==SQLITE_OK ); + assert( !pagerUseWal(pPager) ); + assert( p->eLock>=EXCLUSIVE_LOCK ); + assert( isOpen(p->jfd) + || p->journalMode==PAGER_JOURNALMODE_OFF + || p->journalMode==PAGER_JOURNALMODE_WAL + ); + assert( pPager->dbOrigSize<=pPager->dbHintSize ); + break; + + case PAGER_WRITER_FINISHED: + assert( p->eLock==EXCLUSIVE_LOCK ); + assert( pPager->errCode==SQLITE_OK ); + assert( !pagerUseWal(pPager) ); + assert( isOpen(p->jfd) + || p->journalMode==PAGER_JOURNALMODE_OFF + || p->journalMode==PAGER_JOURNALMODE_WAL + ); + break; + + case PAGER_ERROR: + /* There must be at least one outstanding reference to the pager if + ** in ERROR state. Otherwise the pager should have already dropped + ** back to OPEN state. + */ + assert( pPager->errCode!=SQLITE_OK ); + assert( sqlite3PcacheRefCount(pPager->pPCache)>0 ); + break; + } return 1; } + +/* +** Return a pointer to a human readable string in a static buffer +** containing the state of the Pager object passed as an argument. This +** is intended to be used within debuggers. For example, as an alternative +** to "print *pPager" in gdb: +** +** (gdb) printf "%s", print_pager_state(pPager) +*/ +static char *print_pager_state(Pager *p){ + static char zRet[1024]; + + sqlite3_snprintf(1024, zRet, + "Filename: %s\n" + "State: %s errCode=%d\n" + "Lock: %s\n" + "Locking mode: locking_mode=%s\n" + "Journal mode: journal_mode=%s\n" + "Backing store: tempFile=%d memDb=%d useJournal=%d\n" + "Journal: journalOff=%lld journalHdr=%lld\n" + "Size: dbsize=%d dbOrigSize=%d dbFileSize=%d\n" + , p->zFilename + , p->eState==PAGER_OPEN ? "OPEN" : + p->eState==PAGER_READER ? "READER" : + p->eState==PAGER_WRITER_LOCKED ? "WRITER_LOCKED" : + p->eState==PAGER_WRITER_CACHEMOD ? "WRITER_CACHEMOD" : + p->eState==PAGER_WRITER_DBMOD ? "WRITER_DBMOD" : + p->eState==PAGER_WRITER_FINISHED ? "WRITER_FINISHED" : + p->eState==PAGER_ERROR ? "ERROR" : "?error?" + , (int)p->errCode + , p->eLock==NO_LOCK ? "NO_LOCK" : + p->eLock==RESERVED_LOCK ? "RESERVED" : + p->eLock==EXCLUSIVE_LOCK ? "EXCLUSIVE" : + p->eLock==SHARED_LOCK ? "SHARED" : + p->eLock==UNKNOWN_LOCK ? "UNKNOWN" : "?error?" + , p->exclusiveMode ? "exclusive" : "normal" + , p->journalMode==PAGER_JOURNALMODE_MEMORY ? "memory" : + p->journalMode==PAGER_JOURNALMODE_OFF ? "off" : + p->journalMode==PAGER_JOURNALMODE_DELETE ? "delete" : + p->journalMode==PAGER_JOURNALMODE_PERSIST ? "persist" : + p->journalMode==PAGER_JOURNALMODE_TRUNCATE ? "truncate" : + p->journalMode==PAGER_JOURNALMODE_WAL ? "wal" : "?error?" + , (int)p->tempFile, (int)p->memDb, (int)p->useJournal + , p->journalOff, p->journalHdr + , (int)p->dbSize, (int)p->dbOrigSize, (int)p->dbFileSize + ); + + return zRet; +} #endif /* ** Return true if it is necessary to write page *pPg into the sub-journal. ** A page needs to be written into the sub-journal if there exists one @@ -34584,10 +35596,11 @@ /* ** Write a 32-bit integer into a string buffer in big-endian byte order. */ #define put32bits(A,B) sqlite3Put4byte((u8*)A,B) + /* ** Write a 32-bit integer into the given file descriptor. Return SQLITE_OK ** on success or an error code is something goes wrong. */ @@ -34596,31 +35609,57 @@ put32bits(ac, val); return sqlite3OsWrite(fd, ac, 4, offset); } /* -** The argument to this macro is a file descriptor (type sqlite3_file*). -** Return 0 if it is not open, or non-zero (but not 1) if it is. -** -** This is so that expressions can be written as: -** -** if( isOpen(pPager->jfd) ){ ... -** -** instead of -** -** if( pPager->jfd->pMethods ){ ... -*/ -#define isOpen(pFd) ((pFd)->pMethods) +** Unlock the database file to level eLock, which must be either NO_LOCK +** or SHARED_LOCK. Regardless of whether or not the call to xUnlock() +** succeeds, set the Pager.eLock variable to match the (attempted) new lock. +** +** Except, if Pager.eLock is set to UNKNOWN_LOCK when this function is +** called, do not modify it. See the comment above the #define of +** UNKNOWN_LOCK for an explanation of this. +*/ +static int pagerUnlockDb(Pager *pPager, int eLock){ + int rc = SQLITE_OK; + + assert( !pPager->exclusiveMode ); + assert( eLock==NO_LOCK || eLock==SHARED_LOCK ); + assert( eLock!=NO_LOCK || pagerUseWal(pPager)==0 ); + if( isOpen(pPager->fd) ){ + assert( pPager->eLock>=eLock ); + rc = sqlite3OsUnlock(pPager->fd, eLock); + if( pPager->eLock!=UNKNOWN_LOCK ){ + pPager->eLock = (u8)eLock; + } + IOTRACE(("UNLOCK %p %d\n", pPager, eLock)) + } + return rc; +} /* -** If file pFd is open, call sqlite3OsUnlock() on it. +** Lock the database file to level eLock, which must be either SHARED_LOCK, +** RESERVED_LOCK or EXCLUSIVE_LOCK. If the caller is successful, set the +** Pager.eLock variable to the new locking state. +** +** Except, if Pager.eLock is set to UNKNOWN_LOCK when this function is +** called, do not modify it unless the new locking state is EXCLUSIVE_LOCK. +** See the comment above the #define of UNKNOWN_LOCK for an explanation +** of this. */ -static int osUnlock(sqlite3_file *pFd, int eLock){ - if( !isOpen(pFd) ){ - return SQLITE_OK; +static int pagerLockDb(Pager *pPager, int eLock){ + int rc = SQLITE_OK; + + assert( eLock==SHARED_LOCK || eLock==RESERVED_LOCK || eLock==EXCLUSIVE_LOCK ); + if( pPager->eLock<eLock || pPager->eLock==UNKNOWN_LOCK ){ + rc = sqlite3OsLock(pPager->fd, eLock); + if( rc==SQLITE_OK && (pPager->eLock!=UNKNOWN_LOCK||eLock==EXCLUSIVE_LOCK) ){ + pPager->eLock = (u8)eLock; + IOTRACE(("LOCK %p %d\n", pPager, eLock)) + } } - return sqlite3OsUnlock(pFd, eLock); + return rc; } /* ** This function determines whether or not the atomic-write optimization ** can be used with this pager. The optimization can be used if: @@ -34692,17 +35731,18 @@ ** that the page is either dirty or still matches the calculated page-hash. */ #define CHECK_PAGE(x) checkPage(x) static void checkPage(PgHdr *pPg){ Pager *pPager = pPg->pPager; - assert( !pPg->pageHash || pPager->errCode - || (pPg->flags&PGHDR_DIRTY) || pPg->pageHash==pager_pagehash(pPg) ); + assert( pPager->eState!=PAGER_ERROR ); + assert( (pPg->flags&PGHDR_DIRTY) || pPg->pageHash==pager_pagehash(pPg) ); } #else #define pager_datahash(X,Y) 0 #define pager_pagehash(X) 0 +#define pager_set_pagehash(X) #define CHECK_PAGE(x) #endif /* SQLITE_CHECK_PAGES */ /* ** When this is called the journal file for pager pPager must be open. @@ -34865,11 +35905,11 @@ ** Followed by (JOURNAL_HDR_SZ - 28) bytes of unused space. */ static int writeJournalHdr(Pager *pPager){ int rc = SQLITE_OK; /* Return code */ char *zHeader = pPager->pTmpSpace; /* Temporary space used to build header */ - u32 nHeader = pPager->pageSize; /* Size of buffer pointed to by zHeader */ + u32 nHeader = (u32)pPager->pageSize;/* Size of buffer pointed to by zHeader */ u32 nWrite; /* Bytes of header sector written */ int ii; /* Loop counter */ assert( isOpen(pPager->jfd) ); /* Journal file must be open. */ @@ -34908,11 +35948,11 @@ ** ** * When the SQLITE_IOCAP_SAFE_APPEND flag is set. This guarantees ** that garbage data is never appended to the journal file. */ assert( isOpen(pPager->fd) || pPager->noSync ); - if( (pPager->noSync) || (pPager->journalMode==PAGER_JOURNALMODE_MEMORY) + if( pPager->noSync || (pPager->journalMode==PAGER_JOURNALMODE_MEMORY) || (sqlite3OsDeviceCharacteristics(pPager->fd)&SQLITE_IOCAP_SAFE_APPEND) ){ memcpy(zHeader, aJournalMagic, sizeof(aJournalMagic)); put32bits(&zHeader[sizeof(aJournalMagic)], 0xffffffff); }else{ @@ -35032,18 +36072,25 @@ } if( pPager->journalOff==0 ){ u32 iPageSize; /* Page-size field of journal header */ u32 iSectorSize; /* Sector-size field of journal header */ - u16 iPageSize16; /* Copy of iPageSize in 16-bit variable */ /* Read the page-size and sector-size journal header fields. */ if( SQLITE_OK!=(rc = read32bits(pPager->jfd, iHdrOff+20, &iSectorSize)) || SQLITE_OK!=(rc = read32bits(pPager->jfd, iHdrOff+24, &iPageSize)) ){ return rc; } + + /* Versions of SQLite prior to 3.5.8 set the page-size field of the + ** journal header to zero. In this case, assume that the Pager.pageSize + ** variable is already set to the correct page size. + */ + if( iPageSize==0 ){ + iPageSize = pPager->pageSize; + } /* Check that the values read from the page-size and sector-size fields ** are within range. To be 'in range', both values need to be a power ** of two greater than or equal to 512 or 32, and not greater than their ** respective compile time maximum limits. @@ -35062,14 +36109,12 @@ /* Update the page-size to match the value read from the journal. ** Use a testcase() macro to make sure that malloc failure within ** PagerSetPagesize() is tested. */ - iPageSize16 = (u16)iPageSize; - rc = sqlite3PagerSetPagesize(pPager, &iPageSize16, -1); + rc = sqlite3PagerSetPagesize(pPager, &iPageSize, -1); testcase( rc!=SQLITE_OK ); - assert( rc!=SQLITE_OK || iPageSize16==(u16)iPageSize ); /* Update the assumed sector-size to match the value used by ** the process that created this journal. If this journal was ** created by a process other than this one, then this routine ** is being called from within pager_playback(). The local value @@ -35108,10 +36153,12 @@ i64 iHdrOff; /* Offset of header in journal file */ i64 jrnlSize; /* Size of journal file on disk */ u32 cksum = 0; /* Checksum of string zMaster */ assert( pPager->setMaster==0 ); + assert( !pagerUseWal(pPager) ); + if( !zMaster || pPager->journalMode==PAGER_JOURNALMODE_MEMORY || pPager->journalMode==PAGER_JOURNALMODE_OFF ){ return SQLITE_OK; @@ -35144,11 +36191,10 @@ || (0 != (rc = sqlite3OsWrite(pPager->jfd, aJournalMagic, 8, iHdrOff+4+nMaster+8))) ){ return rc; } pPager->journalOff += (nMaster+20); - pPager->needSync = !pPager->noSync; /* If the pager is in peristent-journal mode, then the physical ** journal-file may extend past the end of the master-journal name ** and 8 bytes of magic data just written to the file. This is ** dangerous because the code to rollback a hot-journal file @@ -35180,21 +36226,15 @@ (void)sqlite3PcacheFetch(pPager->pPCache, pgno, 0, &p); return p; } /* -** Unless the pager is in error-state, discard all in-memory pages. If -** the pager is in error-state, then this call is a no-op. -** -** TODO: Why can we not reset the pager while in error state? +** Discard the entire contents of the in-memory page-cache. */ static void pager_reset(Pager *pPager){ - if( SQLITE_OK==pPager->errCode ){ - sqlite3BackupRestart(pPager->pBackup); - sqlite3PcacheClear(pPager->pPCache); - pPager->dbSizeValid = 0; - } + sqlite3BackupRestart(pPager->pBackup); + sqlite3PcacheClear(pPager->pPCache); } /* ** Free all structures in the Pager.aSavepoint[] array and set both ** Pager.aSavepoint and Pager.nSavepoint to zero. Close the sub-journal @@ -35233,38 +36273,43 @@ } return rc; } /* -** Return true if this pager uses a write-ahead log instead of the usual -** rollback journal. Otherwise false. -*/ -#ifndef SQLITE_OMIT_WAL -static int pagerUseWal(Pager *pPager){ - return (pPager->pWal!=0); -} -#else -# define pagerUseWal(x) 0 -# define pagerRollbackWal(x) 0 -# define pagerWalFrames(v,w,x,y,z) 0 -# define pagerOpenWalIfPresent(z) SQLITE_OK -# define pagerBeginReadTransaction(z) SQLITE_OK -#endif - -/* -** Unlock the database file. This function is a no-op if the pager -** is in exclusive mode. +** This function is a no-op if the pager is in exclusive mode and not +** in the ERROR state. Otherwise, it switches the pager to PAGER_OPEN +** state. ** -** If the pager is currently in error state, discard the contents of -** the cache and reset the Pager structure internal state. If there is -** an open journal-file, then the next time a shared-lock is obtained -** on the pager file (by this or any other process), it will be -** treated as a hot-journal and rolled back. +** If the pager is not in exclusive-access mode, the database file is +** completely unlocked. If the file is unlocked and the file-system does +** not exhibit the UNDELETABLE_WHEN_OPEN property, the journal file is +** closed (if it is open). +** +** If the pager is in ERROR state when this function is called, the +** contents of the pager cache are discarded before switching back to +** the OPEN state. Regardless of whether the pager is in exclusive-mode +** or not, any journal file left in the file-system will be treated +** as a hot-journal and rolled back the next time a read-transaction +** is opened (by this or by any other connection). */ static void pager_unlock(Pager *pPager){ - if( !pPager->exclusiveMode ){ - int rc = SQLITE_OK; /* Return code */ + + assert( pPager->eState==PAGER_READER + || pPager->eState==PAGER_OPEN + || pPager->eState==PAGER_ERROR + ); + + sqlite3BitvecDestroy(pPager->pInJournal); + pPager->pInJournal = 0; + releaseAllSavepoints(pPager); + + if( pagerUseWal(pPager) ){ + assert( !isOpen(pPager->jfd) ); + sqlite3WalEndReadTransaction(pPager->pWal); + pPager->eState = PAGER_OPEN; + }else if( !pPager->exclusiveMode ){ + int rc; /* Error code returned by pagerUnlockDb() */ int iDc = isOpen(pPager->fd)?sqlite3OsDeviceCharacteristics(pPager->fd):0; /* If the operating system support deletion of open files, then ** close the journal file when dropping the database lock. Otherwise ** another connection with journal_mode=delete might delete the file @@ -35280,62 +36325,60 @@ || 1!=(pPager->journalMode & 5) ){ sqlite3OsClose(pPager->jfd); } - sqlite3BitvecDestroy(pPager->pInJournal); - pPager->pInJournal = 0; - releaseAllSavepoints(pPager); - - /* If the file is unlocked, somebody else might change it. The - ** values stored in Pager.dbSize etc. might become invalid if - ** this happens. One can argue that this doesn't need to be cleared - ** until the change-counter check fails in PagerSharedLock(). - ** Clearing the page size cache here is being conservative. - */ - pPager->dbSizeValid = 0; - - if( pagerUseWal(pPager) ){ - sqlite3WalEndReadTransaction(pPager->pWal); - }else{ - rc = osUnlock(pPager->fd, NO_LOCK); - } - if( rc ){ - pPager->errCode = rc; - } - IOTRACE(("UNLOCK %p\n", pPager)) - - /* If Pager.errCode is set, the contents of the pager cache cannot be - ** trusted. Now that the pager file is unlocked, the contents of the - ** cache can be discarded and the error code safely cleared. - */ - if( pPager->errCode ){ - if( rc==SQLITE_OK ){ - pPager->errCode = SQLITE_OK; - } - pager_reset(pPager); - } - + /* If the pager is in the ERROR state and the call to unlock the database + ** file fails, set the current lock to UNKNOWN_LOCK. See the comment + ** above the #define for UNKNOWN_LOCK for an explanation of why this + ** is necessary. + */ + rc = pagerUnlockDb(pPager, NO_LOCK); + if( rc!=SQLITE_OK && pPager->eState==PAGER_ERROR ){ + pPager->eLock = UNKNOWN_LOCK; + } + + /* The pager state may be changed from PAGER_ERROR to PAGER_OPEN here + ** without clearing the error code. This is intentional - the error + ** code is cleared and the cache reset in the block below. + */ + assert( pPager->errCode || pPager->eState!=PAGER_ERROR ); pPager->changeCountDone = 0; - pPager->state = PAGER_UNLOCK; - pPager->dbModified = 0; + pPager->eState = PAGER_OPEN; } + + /* If Pager.errCode is set, the contents of the pager cache cannot be + ** trusted. Now that there are no outstanding references to the pager, + ** it can safely move back to PAGER_OPEN state. This happens in both + ** normal and exclusive-locking mode. + */ + if( pPager->errCode ){ + assert( !MEMDB ); + pager_reset(pPager); + pPager->changeCountDone = pPager->tempFile; + pPager->eState = PAGER_OPEN; + pPager->errCode = SQLITE_OK; + } + + pPager->journalOff = 0; + pPager->journalHdr = 0; + pPager->setMaster = 0; } /* -** This function should be called when an IOERR, CORRUPT or FULL error -** may have occurred. The first argument is a pointer to the pager -** structure, the second the error-code about to be returned by a pager -** API function. The value returned is a copy of the second argument -** to this function. -** -** If the second argument is SQLITE_IOERR, SQLITE_CORRUPT, or SQLITE_FULL -** the error becomes persistent. Until the persistent error is cleared, -** subsequent API calls on this Pager will immediately return the same -** error code. -** -** A persistent error indicates that the contents of the pager-cache +** This function is called whenever an IOERR or FULL error that requires +** the pager to transition into the ERROR state may ahve occurred. +** The first argument is a pointer to the pager structure, the second +** the error-code about to be returned by a pager API function. The +** value returned is a copy of the second argument to this function. +** +** If the second argument is SQLITE_FULL, SQLITE_IOERR or one of the +** IOERR sub-codes, the pager enters the ERROR state and the error code +** is stored in Pager.errCode. While the pager remains in the ERROR state, +** all major API calls on the Pager will immediately return Pager.errCode. +** +** The ERROR state indicates that the contents of the pager-cache ** cannot be trusted. This state can be cleared by completely discarding ** the contents of the pager-cache. If a transaction was active when ** the persistent error occurred, then the rollback journal may need ** to be replayed to restore the contents of the database file (as if ** it were a hot-journal). @@ -35348,49 +36391,25 @@ pPager->errCode==SQLITE_OK || (pPager->errCode & 0xff)==SQLITE_IOERR ); if( rc2==SQLITE_FULL || rc2==SQLITE_IOERR ){ pPager->errCode = rc; + pPager->eState = PAGER_ERROR; } return rc; } -/* -** Execute a rollback if a transaction is active and unlock the -** database file. -** -** If the pager has already entered the error state, do not attempt -** the rollback at this time. Instead, pager_unlock() is called. The -** call to pager_unlock() will discard all in-memory pages, unlock -** the database file and clear the error state. If this means that -** there is a hot-journal left in the file-system, the next connection -** to obtain a shared lock on the pager (which may be this one) will -** roll it back. -** -** If the pager has not already entered the error state, but an IO or -** malloc error occurs during a rollback, then this will itself cause -** the pager to enter the error state. Which will be cleared by the -** call to pager_unlock(), as described above. -*/ -static void pagerUnlockAndRollback(Pager *pPager){ - if( pPager->errCode==SQLITE_OK && pPager->state>=PAGER_RESERVED ){ - sqlite3BeginBenignMalloc(); - sqlite3PagerRollback(pPager); - sqlite3EndBenignMalloc(); - } - pager_unlock(pPager); -} - /* ** This routine ends a transaction. A transaction is usually ended by ** either a COMMIT or a ROLLBACK operation. This routine may be called ** after rollback of a hot-journal, or if an error occurs while opening ** the journal file or writing the very first journal-header of a ** database transaction. ** -** If the pager is in PAGER_SHARED or PAGER_UNLOCK state when this -** routine is called, it is a no-op (returns SQLITE_OK). +** This routine is never called in PAGER_ERROR state. If it is called +** in PAGER_NONE or PAGER_SHARED state and the lock held is less +** exclusive than a RESERVED lock, it is a no-op. ** ** Otherwise, any active savepoints are released. ** ** If the journal file is open, then it is "finalized". Once a journal ** file has been finalized it is not possible to use it to roll back a @@ -35417,17 +36436,13 @@ ** If the pager is running in exclusive mode, this method of finalizing ** the journal file is never used. Instead, if the journalMode is ** DELETE and the pager is in exclusive mode, the method described under ** journalMode==PERSIST is used instead. ** -** After the journal is finalized, if running in non-exclusive mode, the -** pager moves to PAGER_SHARED state (and downgrades the lock on the -** database file accordingly). -** -** If the pager is running in exclusive mode and is in PAGER_SYNCED state, -** it moves to PAGER_EXCLUSIVE. No locks are downgraded when running in -** exclusive mode. +** After the journal is finalized, the pager moves to PAGER_READER state. +** If running in non-exclusive rollback mode, the lock on the file is +** downgraded to a SHARED_LOCK. ** ** SQLITE_OK is returned if no error occurs. If an error occurs during ** any of the IO operations to finalize the journal file or unlock the ** database then the IO error code is returned to the user. If the ** operation to finalize the journal file fails, then the code still @@ -35438,15 +36453,30 @@ */ static int pager_end_transaction(Pager *pPager, int hasMaster){ int rc = SQLITE_OK; /* Error code from journal finalization operation */ int rc2 = SQLITE_OK; /* Error code from db file unlock operation */ - if( pPager->state<PAGER_RESERVED ){ + /* Do nothing if the pager does not have an open write transaction + ** or at least a RESERVED lock. This function may be called when there + ** is no write-transaction active but a RESERVED or greater lock is + ** held under two circumstances: + ** + ** 1. After a successful hot-journal rollback, it is called with + ** eState==PAGER_NONE and eLock==EXCLUSIVE_LOCK. + ** + ** 2. If a connection with locking_mode=exclusive holding an EXCLUSIVE + ** lock switches back to locking_mode=normal and then executes a + ** read-transaction, this function is called with eState==PAGER_READER + ** and eLock==EXCLUSIVE_LOCK when the read-transaction is closed. + */ + assert( assert_pager_state(pPager) ); + assert( pPager->eState!=PAGER_ERROR ); + if( pPager->eState<PAGER_WRITER_LOCKED && pPager->eLock<RESERVED_LOCK ){ return SQLITE_OK; } + releaseAllSavepoints(pPager); - assert( isOpen(pPager->jfd) || pPager->pInJournal==0 ); if( isOpen(pPager->jfd) ){ assert( !pagerUseWal(pPager) ); /* Finalize the journal file. */ @@ -35458,18 +36488,15 @@ rc = SQLITE_OK; }else{ rc = sqlite3OsTruncate(pPager->jfd, 0); } pPager->journalOff = 0; - pPager->journalStarted = 0; }else if( pPager->journalMode==PAGER_JOURNALMODE_PERSIST || (pPager->exclusiveMode && pPager->journalMode!=PAGER_JOURNALMODE_WAL) ){ rc = zeroJournalHdr(pPager, hasMaster); - pager_error(pPager, rc); pPager->journalOff = 0; - pPager->journalStarted = 0; }else{ /* This branch may be executed with Pager.journalMode==MEMORY if ** a hot-journal was just rolled back. In this case the journal ** file should be closed and deleted. If this connection writes to ** the database file, it will do so using an in-memory journal. @@ -35481,52 +36508,80 @@ sqlite3OsClose(pPager->jfd); if( !pPager->tempFile ){ rc = sqlite3OsDelete(pPager->pVfs, pPager->zJournal, 0); } } + } #ifdef SQLITE_CHECK_PAGES - sqlite3PcacheIterateDirty(pPager->pPCache, pager_set_pagehash); + sqlite3PcacheIterateDirty(pPager->pPCache, pager_set_pagehash); + if( pPager->dbSize==0 && sqlite3PcacheRefCount(pPager->pPCache)>0 ){ + PgHdr *p = pager_lookup(pPager, 1); + if( p ){ + p->pageHash = 0; + sqlite3PagerUnref(p); + } + } #endif - } + sqlite3BitvecDestroy(pPager->pInJournal); pPager->pInJournal = 0; pPager->nRec = 0; sqlite3PcacheCleanAll(pPager->pPCache); + sqlite3PcacheTruncate(pPager->pPCache, pPager->dbSize); if( pagerUseWal(pPager) ){ + /* Drop the WAL write-lock, if any. Also, if the connection was in + ** locking_mode=exclusive mode but is no longer, drop the EXCLUSIVE + ** lock held on the database file. + */ rc2 = sqlite3WalEndWriteTransaction(pPager->pWal); assert( rc2==SQLITE_OK ); - pPager->state = PAGER_SHARED; - - /* If the connection was in locking_mode=exclusive mode but is no longer, - ** drop the EXCLUSIVE lock held on the database file. - */ - if( !pPager->exclusiveMode && sqlite3WalExclusiveMode(pPager->pWal, 0) ){ - rc2 = osUnlock(pPager->fd, SHARED_LOCK); - } - }else if( !pPager->exclusiveMode ){ - rc2 = osUnlock(pPager->fd, SHARED_LOCK); - pPager->state = PAGER_SHARED; + } + if( !pPager->exclusiveMode + && (!pagerUseWal(pPager) || sqlite3WalExclusiveMode(pPager->pWal, 0)) + ){ + rc2 = pagerUnlockDb(pPager, SHARED_LOCK); pPager->changeCountDone = 0; - }else if( pPager->state==PAGER_SYNCED ){ - pPager->state = PAGER_EXCLUSIVE; - } - pPager->setMaster = 0; - pPager->needSync = 0; - pPager->dbModified = 0; - - /* TODO: Is this optimal? Why is the db size invalidated here - ** when the database file is not unlocked? */ - pPager->dbOrigSize = 0; - sqlite3PcacheTruncate(pPager->pPCache, pPager->dbSize); - if( !MEMDB ){ - pPager->dbSizeValid = 0; - } + } + pPager->eState = PAGER_READER; + pPager->setMaster = 0; return (rc==SQLITE_OK?rc2:rc); } + +/* +** Execute a rollback if a transaction is active and unlock the +** database file. +** +** If the pager has already entered the ERROR state, do not attempt +** the rollback at this time. Instead, pager_unlock() is called. The +** call to pager_unlock() will discard all in-memory pages, unlock +** the database file and move the pager back to OPEN state. If this +** means that there is a hot-journal left in the file-system, the next +** connection to obtain a shared lock on the pager (which may be this one) +** will roll it back. +** +** If the pager has not already entered the ERROR state, but an IO or +** malloc error occurs during a rollback, then this will itself cause +** the pager to enter the ERROR state. Which will be cleared by the +** call to pager_unlock(), as described above. +*/ +static void pagerUnlockAndRollback(Pager *pPager){ + if( pPager->eState!=PAGER_ERROR && pPager->eState!=PAGER_OPEN ){ + assert( assert_pager_state(pPager) ); + if( pPager->eState>=PAGER_WRITER_LOCKED ){ + sqlite3BeginBenignMalloc(); + sqlite3PagerRollback(pPager); + sqlite3EndBenignMalloc(); + }else if( !pPager->exclusiveMode ){ + assert( pPager->eState==PAGER_READER ); + pager_end_transaction(pPager, 0); + } + } + pager_unlock(pPager); +} /* ** Parameter aData must point to a buffer of pPager->pageSize bytes ** of data. Compute and return a checksum based ont the contents of the ** page of data and the current value of pPager->cksumInit. @@ -35574,13 +36629,12 @@ ** Read a single page from either the journal file (if isMainJrnl==1) or ** from the sub-journal (if isMainJrnl==0) and playback that page. ** The page begins at offset *pOffset into the file. The *pOffset ** value is increased to the start of the next page in the journal. ** -** The isMainJrnl flag is true if this is the main rollback journal and -** false for the statement journal. The main rollback journal uses -** checksums - the statement journal does not. +** The main rollback journal uses checksums - the statement journal does +** not. ** ** If the page number of the page record read from the (sub-)journal file ** is greater than the current value of Pager.dbSize, then playback is ** skipped and SQLITE_OK is returned. ** @@ -35629,10 +36683,21 @@ assert( isSavepnt || pDone==0 ); /* pDone never used on non-savepoint */ aData = pPager->pTmpSpace; assert( aData ); /* Temp storage must have already been allocated */ assert( pagerUseWal(pPager)==0 || (!isMainJrnl && isSavepnt) ); + + /* Either the state is greater than PAGER_WRITER_CACHEMOD (a transaction + ** or savepoint rollback done at the request of the caller) or this is + ** a hot-journal rollback. If it is a hot-journal rollback, the pager + ** is in state OPEN and holds an EXCLUSIVE lock. Hot-journal rollback + ** only reads from the main journal, not the sub-journal. + */ + assert( pPager->eState>=PAGER_WRITER_CACHEMOD + || (pPager->eState==PAGER_OPEN && pPager->eLock==EXCLUSIVE_LOCK) + ); + assert( pPager->eState>=PAGER_WRITER_CACHEMOD || isMainJrnl ); /* Read the page number and page data from the journal or sub-journal ** file. Return an error code to the caller if an IO error occurs. */ jfd = isMainJrnl ? pPager->jfd : pPager->sjfd; @@ -35666,20 +36731,19 @@ ** rollback, then don't bother to play it back again. */ if( pDone && (rc = sqlite3BitvecSet(pDone, pgno))!=SQLITE_OK ){ return rc; } - assert( pPager->state==PAGER_RESERVED || pPager->state>=PAGER_EXCLUSIVE ); /* When playing back page 1, restore the nReserve setting */ if( pgno==1 && pPager->nReserve!=((u8*)aData)[20] ){ pPager->nReserve = ((u8*)aData)[20]; pagerReportSize(pPager); } - /* If the pager is in RESERVED state, then there must be a copy of this + /* If the pager is in CACHEMOD state, then there must be a copy of this ** page in the pager cache. In this case just update the pager cache, ** not the database file. The page is left marked dirty in this case. ** ** An exception to the above rule: If the database is in no-sync mode ** and a page is moved during an incremental vacuum then the page may @@ -35686,12 +36750,15 @@ ** not be in the pager cache. Later: if a malloc() or IO error occurs ** during a Movepage() call, then the page may not be in the cache ** either. So the condition described in the above paragraph is not ** assert()able. ** - ** If in EXCLUSIVE state, then we update the pager cache if it exists - ** and the main file. The page is then marked not dirty. + ** If in WRITER_DBMOD, WRITER_FINISHED or OPEN state, then we update the + ** pager cache if it exists and the main file. The page is then marked + ** not dirty. Since this code is only executed in PAGER_OPEN state for + ** a hot-journal rollback, it is guaranteed that the page-cache is empty + ** if the pager is in OPEN state. ** ** Ticket #1171: The statement journal might contain page content that is ** different from the page content at the start of the transaction. ** This occurs when a page is changed prior to the start of a statement ** then changed again within the statement. When rolling back such a @@ -35713,21 +36780,22 @@ pPg = 0; }else{ pPg = pager_lookup(pPager, pgno); } assert( pPg || !MEMDB ); + assert( pPager->eState!=PAGER_OPEN || pPg==0 ); PAGERTRACE(("PLAYBACK %d page %d hash(%08x) %s\n", PAGERID(pPager), pgno, pager_datahash(pPager->pageSize, (u8*)aData), (isMainJrnl?"main-journal":"sub-journal") )); if( isMainJrnl ){ isSynced = pPager->noSync || (*pOffset <= pPager->journalHdr); }else{ isSynced = (pPg==0 || 0==(pPg->flags & PGHDR_NEED_SYNC)); } - if( (pPager->state>=PAGER_EXCLUSIVE) - && isOpen(pPager->fd) + if( isOpen(pPager->fd) + && (pPager->eState>=PAGER_WRITER_DBMOD || pPager->eState==PAGER_OPEN) && isSynced ){ i64 ofst = (pgno-1)*(i64)pPager->pageSize; testcase( !isSavepnt && pPg!=0 && (pPg->flags&PGHDR_NEED_SYNC)!=0 ); assert( !pagerUseWal(pPager) ); @@ -35799,13 +36867,12 @@ ** database corruption may ensue. */ assert( !pagerUseWal(pPager) ); sqlite3PcacheMakeClean(pPg); } -#ifdef SQLITE_CHECK_PAGES - pPg->pageHash = pager_pagehash(pPg); -#endif + pager_set_pagehash(pPg); + /* If this was page 1, then restore the value of Pager.dbFileVers. ** Do this before any decoding. */ if( pgno==1 ){ memcpy(&pPager->dbFileVers, &((u8*)pData)[24],sizeof(pPager->dbFileVers)); } @@ -35953,14 +37020,14 @@ /* ** This function is used to change the actual size of the database ** file in the file-system. This only happens when committing a transaction, ** or rolling back a transaction (including rolling back a hot-journal). ** -** If the main database file is not open, or an exclusive lock is not -** held, this function is a no-op. Otherwise, the size of the file is -** changed to nPage pages (nPage*pPager->pageSize bytes). If the file -** on disk is currently larger than nPage pages, then use the VFS +** If the main database file is not open, or the pager is not in either +** DBMOD or OPEN state, this function is a no-op. Otherwise, the size +** of the file is changed to nPage pages (nPage*pPager->pageSize bytes). +** If the file on disk is currently larger than nPage pages, then use the VFS ** xTruncate() method to truncate it. ** ** Or, it might might be the case that the file on disk is smaller than ** nPage pages. Some operating system implementations can get confused if ** you try to truncate a file to some size that is larger than it @@ -35970,12 +37037,18 @@ ** If successful, return SQLITE_OK. If an IO error occurs while modifying ** the database file, return the error code to the caller. */ static int pager_truncate(Pager *pPager, Pgno nPage){ int rc = SQLITE_OK; - if( pPager->state>=PAGER_EXCLUSIVE && isOpen(pPager->fd) ){ + assert( pPager->eState!=PAGER_ERROR ); + assert( pPager->eState!=PAGER_READER ); + + if( isOpen(pPager->fd) + && (pPager->eState>=PAGER_WRITER_DBMOD || pPager->eState==PAGER_OPEN) + ){ i64 currentSize, newSize; + assert( pPager->eLock==EXCLUSIVE_LOCK ); /* TODO: Is it safe to use Pager.dbFileSize here? */ rc = sqlite3OsFileSize(pPager->fd, ¤tSize); newSize = pPager->pageSize*(i64)nPage; if( rc==SQLITE_OK && currentSize!=newSize ){ if( currentSize>newSize ){ @@ -36095,11 +37168,11 @@ /* Figure out how many records are in the journal. Abort early if ** the journal is empty. */ assert( isOpen(pPager->jfd) ); rc = sqlite3OsFileSize(pPager->jfd, &szJ); - if( rc!=SQLITE_OK || szJ==0 ){ + if( rc!=SQLITE_OK ){ goto end_playback; } /* Read the master journal name from the journal, if it is present. ** If a master journal file name is specified, but the file is not @@ -36129,11 +37202,11 @@ ** occurs. */ while( 1 ){ /* Read the next journal header from the journal file. If there are ** not enough bytes left in the journal file for a complete header, or - ** it is corrupted, then a process must of failed while writing it. + ** it is corrupted, then a process must have failed while writing it. ** This indicates nothing more needs to be rolled back. */ rc = readJournalHdr(pPager, isHot, szJ, &nRec, &mxPg); if( rc!=SQLITE_OK ){ if( rc==SQLITE_DONE ){ @@ -36243,14 +37316,13 @@ if( rc==SQLITE_OK ){ zMaster = pPager->pTmpSpace; rc = readMasterJournal(pPager->jfd, zMaster, pPager->pVfs->mxPathname+1); testcase( rc!=SQLITE_OK ); } - if( rc==SQLITE_OK && pPager->noSync==0 && pPager->state>=PAGER_EXCLUSIVE ){ - rc = sqlite3OsSync(pPager->fd, pPager->sync_flags); - } - if( rc==SQLITE_OK && pPager->noSync==0 && pPager->state>=PAGER_EXCLUSIVE ){ + if( rc==SQLITE_OK && !pPager->noSync + && (pPager->eState>=PAGER_WRITER_DBMOD || pPager->eState==PAGER_OPEN) + ){ rc = sqlite3OsSync(pPager->fd, pPager->sync_flags); } if( rc==SQLITE_OK ){ rc = pager_end_transaction(pPager, zMaster[0]!='\0'); testcase( rc!=SQLITE_OK ); @@ -36288,11 +37360,11 @@ Pgno pgno = pPg->pgno; /* Page number to read */ int rc = SQLITE_OK; /* Return code */ int isInWal = 0; /* True if page is in log file */ int pgsz = pPager->pageSize; /* Number of bytes to read */ - assert( pPager->state>=PAGER_SHARED && !MEMDB ); + assert( pPager->eState>=PAGER_READER && !MEMDB ); assert( isOpen(pPager->fd) ); if( NEVER(!isOpen(pPager->fd)) ){ assert( pPager->tempFile ); memset(pPg->pData, 0, pPager->pageSize); @@ -36435,10 +37507,18 @@ PgHdr *p; for(p=pList; p; p=p->pDirty){ sqlite3BackupUpdate(pPager->pBackup, p->pgno, (u8 *)p->pData); } } + +#ifdef SQLITE_CHECK_PAGES + { + PgHdr *p; + for(p=pList; p; p=p->pDirty) pager_set_pagehash(p); + } +#endif + return rc; } /* ** Begin a read transaction on the WAL. @@ -36451,32 +37531,84 @@ static int pagerBeginReadTransaction(Pager *pPager){ int rc; /* Return code */ int changed = 0; /* True if cache must be reset */ assert( pagerUseWal(pPager) ); + assert( pPager->eState==PAGER_OPEN || pPager->eState==PAGER_READER ); /* sqlite3WalEndReadTransaction() was not called for the previous ** transaction in locking_mode=EXCLUSIVE. So call it now. If we ** are in locking_mode=NORMAL and EndRead() was previously called, ** the duplicate call is harmless. */ sqlite3WalEndReadTransaction(pPager->pWal); rc = sqlite3WalBeginReadTransaction(pPager->pWal, &changed); - if( rc==SQLITE_OK ){ - int dummy; - if( changed ){ - pager_reset(pPager); - assert( pPager->errCode || pPager->dbSizeValid==0 ); - } - rc = sqlite3PagerPagecount(pPager, &dummy); - } - pPager->state = PAGER_SHARED; + if( rc!=SQLITE_OK || changed ){ + pager_reset(pPager); + } return rc; } +#endif +/* +** This function is called as part of the transition from PAGER_OPEN +** to PAGER_READER state to determine the size of the database file +** in pages (assuming the page size currently stored in Pager.pageSize). +** +** If no error occurs, SQLITE_OK is returned and the size of the database +** in pages is stored in *pnPage. Otherwise, an error code (perhaps +** SQLITE_IOERR_FSTAT) is returned and *pnPage is left unmodified. +*/ +static int pagerPagecount(Pager *pPager, Pgno *pnPage){ + Pgno nPage; /* Value to return via *pnPage */ + + /* Query the WAL sub-system for the database size. The WalDbsize() + ** function returns zero if the WAL is not open (i.e. Pager.pWal==0), or + ** if the database size is not available. The database size is not + ** available from the WAL sub-system if the log file is empty or + ** contains no valid committed transactions. + */ + assert( pPager->eState==PAGER_OPEN ); + assert( pPager->eLock>=SHARED_LOCK || pPager->noReadlock ); + nPage = sqlite3WalDbsize(pPager->pWal); + + /* If the database size was not available from the WAL sub-system, + ** determine it based on the size of the database file. If the size + ** of the database file is not an integer multiple of the page-size, + ** round down to the nearest page. Except, any file larger than 0 + ** bytes in size is considered to contain at least one page. + */ + if( nPage==0 ){ + i64 n = 0; /* Size of db file in bytes */ + assert( isOpen(pPager->fd) || pPager->tempFile ); + if( isOpen(pPager->fd) ){ + int rc = sqlite3OsFileSize(pPager->fd, &n); + if( rc!=SQLITE_OK ){ + return rc; + } + } + nPage = (Pgno)(n / pPager->pageSize); + if( nPage==0 && n>0 ){ + nPage = 1; + } + } + + /* If the current number of pages in the file is greater than the + ** configured maximum pager number, increase the allowed limit so + ** that the file can be read. + */ + if( nPage>pPager->mxPgno ){ + pPager->mxPgno = (Pgno)nPage; + } + + *pnPage = nPage; + return SQLITE_OK; +} + +#ifndef SQLITE_OMIT_WAL /* ** Check if the *-wal file that corresponds to the database opened by pPager ** exists if the database is not empy, or verify that the *-wal file does ** not exist (by deleting it) if the database file is empty. ** @@ -36485,25 +37617,26 @@ ** if no error occurs, make sure Pager.journalMode is not set to ** PAGER_JOURNALMODE_WAL. ** ** Return SQLITE_OK or an error code. ** -** If the WAL file is opened, also open a snapshot (read transaction). -** ** The caller must hold a SHARED lock on the database file to call this ** function. Because an EXCLUSIVE lock on the db file is required to delete ** a WAL on a none-empty database, this ensures there is no race condition ** between the xAccess() below and an xDelete() being executed by some ** other connection. */ static int pagerOpenWalIfPresent(Pager *pPager){ int rc = SQLITE_OK; + assert( pPager->eState==PAGER_OPEN ); + assert( pPager->eLock>=SHARED_LOCK || pPager->noReadlock ); + if( !pPager->tempFile ){ int isWal; /* True if WAL file exists */ - int nPage; /* Size of the database file */ - assert( pPager->state>=SHARED_LOCK ); - rc = sqlite3PagerPagecount(pPager, &nPage); + Pgno nPage; /* Size of the database file */ + + rc = pagerPagecount(pPager, &nPage); if( rc ) return rc; if( nPage==0 ){ rc = sqlite3OsDelete(pPager->pVfs, pPager->zWal, 0); isWal = 0; }else{ @@ -36511,15 +37644,12 @@ pPager->pVfs, pPager->zWal, SQLITE_ACCESS_EXISTS, &isWal ); } if( rc==SQLITE_OK ){ if( isWal ){ - pager_reset(pPager); + testcase( sqlite3PcachePagecount(pPager->pPCache)==0 ); rc = sqlite3PagerOpenWal(pPager, 0); - if( rc==SQLITE_OK ){ - rc = pagerBeginReadTransaction(pPager); - } }else if( pPager->journalMode==PAGER_JOURNALMODE_WAL ){ pPager->journalMode = PAGER_JOURNALMODE_DELETE; } } } @@ -36567,11 +37697,12 @@ i64 szJ; /* Effective size of the main journal */ i64 iHdrOff; /* End of first segment of main-journal records */ int rc = SQLITE_OK; /* Return code */ Bitvec *pDone = 0; /* Bitvec to ensure pages played back only once */ - assert( pPager->state>=PAGER_SHARED ); + assert( pPager->eState!=PAGER_ERROR ); + assert( pPager->eState>=PAGER_WRITER_LOCKED ); /* Allocate a bitvec to use to store the set of pages rolled back */ if( pSavepoint ){ pDone = sqlite3BitvecCreate(pSavepoint->nOrig); if( !pDone ){ @@ -36706,11 +37837,10 @@ #ifndef SQLITE_OMIT_PAGER_PRAGMAS SQLITE_PRIVATE void sqlite3PagerSetSafetyLevel(Pager *pPager, int level, int bFullFsync){ pPager->noSync = (level==1 || pPager->tempFile) ?1:0; pPager->fullSync = (level==3 && !pPager->tempFile) ?1:0; pPager->sync_flags = (bFullFsync?SQLITE_SYNC_FULL:SQLITE_SYNC_NORMAL); - if( pPager->noSync ) pPager->needSync = 0; } #endif /* ** The following global variable is incremented whenever the library @@ -36788,11 +37918,11 @@ ** Change the page size used by the Pager object. The new page size ** is passed in *pPageSize. ** ** If the pager is in the error state when this function is called, it ** is a no-op. The value returned is the error state error code (i.e. -** one of SQLITE_IOERR, SQLITE_CORRUPT or SQLITE_FULL). +** one of SQLITE_IOERR, an SQLITE_IOERR_xxx sub-code or SQLITE_FULL). ** ** Otherwise, if all of the following are true: ** ** * the new page size (value of *pPageSize) is valid (a power ** of two between 512 and SQLITE_MAX_PAGE_SIZE, inclusive), and @@ -36812,32 +37942,52 @@ ** If the page size is not changed, either because one of the enumerated ** conditions above is not true, the pager was in error state when this ** function was called, or because the memory allocation attempt failed, ** then *pPageSize is set to the old, retained page size before returning. */ -SQLITE_PRIVATE int sqlite3PagerSetPagesize(Pager *pPager, u16 *pPageSize, int nReserve){ - int rc = pPager->errCode; - - if( rc==SQLITE_OK ){ - u16 pageSize = *pPageSize; - assert( pageSize==0 || (pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE) ); - if( (pPager->memDb==0 || pPager->dbSize==0) - && sqlite3PcacheRefCount(pPager->pPCache)==0 - && pageSize && pageSize!=pPager->pageSize - ){ - char *pNew = (char *)sqlite3PageMalloc(pageSize); - if( !pNew ){ - rc = SQLITE_NOMEM; - }else{ - pager_reset(pPager); - pPager->pageSize = pageSize; - sqlite3PageFree(pPager->pTmpSpace); - pPager->pTmpSpace = pNew; - sqlite3PcacheSetPageSize(pPager->pPCache, pageSize); - } - } - *pPageSize = (u16)pPager->pageSize; +SQLITE_PRIVATE int sqlite3PagerSetPagesize(Pager *pPager, u32 *pPageSize, int nReserve){ + int rc = SQLITE_OK; + + /* It is not possible to do a full assert_pager_state() here, as this + ** function may be called from within PagerOpen(), before the state + ** of the Pager object is internally consistent. + ** + ** At one point this function returned an error if the pager was in + ** PAGER_ERROR state. But since PAGER_ERROR state guarantees that + ** there is at least one outstanding page reference, this function + ** is a no-op for that case anyhow. + */ + + u32 pageSize = *pPageSize; + assert( pageSize==0 || (pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE) ); + if( (pPager->memDb==0 || pPager->dbSize==0) + && sqlite3PcacheRefCount(pPager->pPCache)==0 + && pageSize && pageSize!=(u32)pPager->pageSize + ){ + char *pNew = NULL; /* New temp space */ + i64 nByte = 0; + + if( pPager->eState>PAGER_OPEN && isOpen(pPager->fd) ){ + rc = sqlite3OsFileSize(pPager->fd, &nByte); + } + if( rc==SQLITE_OK ){ + pNew = (char *)sqlite3PageMalloc(pageSize); + if( !pNew ) rc = SQLITE_NOMEM; + } + + if( rc==SQLITE_OK ){ + pager_reset(pPager); + pPager->dbSize = (Pgno)(nByte/pageSize); + pPager->pageSize = pageSize; + sqlite3PageFree(pPager->pTmpSpace); + pPager->pTmpSpace = pNew; + sqlite3PcacheSetPageSize(pPager->pPCache, pageSize); + } + } + + *pPageSize = pPager->pageSize; + if( rc==SQLITE_OK ){ if( nReserve<0 ) nReserve = pPager->nReserve; assert( nReserve>=0 && nReserve<1000 ); pPager->nReserve = (i16)nReserve; pagerReportSize(pPager); } @@ -36862,17 +38012,15 @@ ** maximum page count below the current size of the database. ** ** Regardless of mxPage, return the current maximum page count. */ SQLITE_PRIVATE int sqlite3PagerMaxPageCount(Pager *pPager, int mxPage){ - int nPage; if( mxPage>0 ){ pPager->mxPgno = mxPage; } - if( pPager->state!=PAGER_UNLOCK ){ - sqlite3PagerPagecount(pPager, &nPage); - assert( (int)pPager->mxPgno>=nPage ); + if( pPager->eState!=PAGER_OPEN && pPager->mxPgno<pPager->dbSize ){ + pPager->mxPgno = pPager->dbSize; } return pPager->mxPgno; } /* @@ -36933,70 +38081,20 @@ } return rc; } /* -** Return the total number of pages in the database file associated -** with pPager. Normally, this is calculated as (<db file size>/<page-size>). +** This function may only be called when a read-transaction is open on +** the pager. It returns the total number of pages in the database. +** ** However, if the file is between 1 and <page-size> bytes in size, then ** this is considered a 1 page file. -** -** If the pager is in error state when this function is called, then the -** error state error code is returned and *pnPage left unchanged. Or, -** if the file system has to be queried for the size of the file and -** the query attempt returns an IO error, the IO error code is returned -** and *pnPage is left unchanged. -** -** Otherwise, if everything is successful, then SQLITE_OK is returned -** and *pnPage is set to the number of pages in the database. -*/ -SQLITE_PRIVATE int sqlite3PagerPagecount(Pager *pPager, int *pnPage){ - Pgno nPage = 0; /* Value to return via *pnPage */ - - /* Determine the number of pages in the file. Store this in nPage. */ - if( pPager->dbSizeValid ){ - nPage = pPager->dbSize; - }else{ - int rc; /* Error returned by OsFileSize() */ - i64 n = 0; /* File size in bytes returned by OsFileSize() */ - - if( pagerUseWal(pPager) && pPager->state!=PAGER_UNLOCK ){ - sqlite3WalDbsize(pPager->pWal, &nPage); - } - - if( nPage==0 ){ - assert( isOpen(pPager->fd) || pPager->tempFile ); - if( isOpen(pPager->fd) ){ - if( SQLITE_OK!=(rc = sqlite3OsFileSize(pPager->fd, &n)) ){ - pager_error(pPager, rc); - return rc; - } - } - if( n>0 && n<pPager->pageSize ){ - nPage = 1; - }else{ - nPage = (Pgno)(n / pPager->pageSize); - } - } - if( pPager->state!=PAGER_UNLOCK ){ - pPager->dbSize = nPage; - pPager->dbFileSize = nPage; - pPager->dbSizeValid = 1; - } - } - - /* If the current number of pages in the file is greater than the - ** configured maximum pager number, increase the allowed limit so - ** that the file can be read. - */ - if( nPage>pPager->mxPgno ){ - pPager->mxPgno = (Pgno)nPage; - } - - /* Set the output variable and return SQLITE_OK */ - *pnPage = nPage; - return SQLITE_OK; +*/ +SQLITE_PRIVATE void sqlite3PagerPagecount(Pager *pPager, int *pnPage){ + assert( pPager->eState>=PAGER_READER ); + assert( pPager->eState!=PAGER_WRITER_FINISHED ); + *pnPage = (int)pPager->dbSize; } /* ** Try to obtain a lock of type locktype on the database file. If @@ -37013,42 +38111,23 @@ ** variable to locktype before returning. */ static int pager_wait_on_lock(Pager *pPager, int locktype){ int rc; /* Return code */ - /* The OS lock values must be the same as the Pager lock values */ - assert( PAGER_SHARED==SHARED_LOCK ); - assert( PAGER_RESERVED==RESERVED_LOCK ); - assert( PAGER_EXCLUSIVE==EXCLUSIVE_LOCK ); - - /* If the file is currently unlocked then the size must be unknown. It - ** must not have been modified at this point. - */ - assert( pPager->state>=PAGER_SHARED || pPager->dbSizeValid==0 ); - assert( pPager->state>=PAGER_SHARED || pPager->dbModified==0 ); - /* Check that this is either a no-op (because the requested lock is ** already held, or one of the transistions that the busy-handler ** may be invoked during, according to the comment above ** sqlite3PagerSetBusyhandler(). */ - assert( (pPager->state>=locktype) - || (pPager->state==PAGER_UNLOCK && locktype==PAGER_SHARED) - || (pPager->state==PAGER_RESERVED && locktype==PAGER_EXCLUSIVE) + assert( (pPager->eLock>=locktype) + || (pPager->eLock==NO_LOCK && locktype==SHARED_LOCK) + || (pPager->eLock==RESERVED_LOCK && locktype==EXCLUSIVE_LOCK) ); - if( pPager->state>=locktype ){ - rc = SQLITE_OK; - }else{ - do { - rc = sqlite3OsLock(pPager->fd, locktype); - }while( rc==SQLITE_BUSY && pPager->xBusyHandler(pPager->pBusyHandlerArg) ); - if( rc==SQLITE_OK ){ - pPager->state = (u8)locktype; - IOTRACE(("LOCK %p %d\n", pPager, locktype)) - } - } + do { + rc = pagerLockDb(pPager, locktype); + }while( rc==SQLITE_BUSY && pPager->xBusyHandler(pPager->pBusyHandlerArg) ); return rc; } /* ** Function assertTruncateConstraint(pPager) checks that one of the @@ -37089,13 +38168,12 @@ ** function does not actually modify the database file on disk. It ** just sets the internal state of the pager object so that the ** truncation will be done when the current transaction is committed. */ SQLITE_PRIVATE void sqlite3PagerTruncateImage(Pager *pPager, Pgno nPage){ - assert( pPager->dbSizeValid ); assert( pPager->dbSize>=nPage ); - assert( pPager->state>=PAGER_RESERVED ); + assert( pPager->eState>=PAGER_WRITER_CACHEMOD ); pPager->dbSize = nPage; assertTruncateConstraint(pPager); } @@ -37141,11 +38219,11 @@ SQLITE_PRIVATE int sqlite3PagerClose(Pager *pPager){ u8 *pTmp = (u8 *)pPager->pTmpSpace; disable_simulated_io_errors(); sqlite3BeginBenignMalloc(); - pPager->errCode = 0; + /* pPager->errCode = 0; */ pPager->exclusiveMode = 0; #ifndef SQLITE_OMIT_WAL sqlite3WalClose(pPager->pWal, (pPager->noSync ? 0 : pPager->sync_flags), pPager->pageSize, pTmp @@ -37154,18 +38232,23 @@ #endif pager_reset(pPager); if( MEMDB ){ pager_unlock(pPager); }else{ - /* Set Pager.journalHdr to -1 for the benefit of the pager_playback() - ** call which may be made from within pagerUnlockAndRollback(). If it - ** is not -1, then the unsynced portion of an open journal file may - ** be played back into the database. If a power failure occurs while - ** this is happening, the database may become corrupt. + /* If it is open, sync the journal file before calling UnlockAndRollback. + ** If this is not done, then an unsynced portion of the open journal + ** file may be played back into the database. If a power failure occurs + ** while this is happening, the database could become corrupt. + ** + ** If an error occurs while trying to sync the journal, shift the pager + ** into the ERROR state. This causes UnlockAndRollback to unlock the + ** database and close the journal file without attempting to roll it + ** back or finalize it. The next database user will have to do hot-journal + ** rollback before accessing the database file. */ if( isOpen(pPager->jfd) ){ - pPager->errCode = pagerSyncHotJournal(pPager); + pager_error(pPager, pagerSyncHotJournal(pPager)); } pagerUnlockAndRollback(pPager); } sqlite3EndBenignMalloc(); enable_simulated_io_errors(); @@ -37206,13 +38289,13 @@ /* ** Sync the journal. In other words, make sure all the pages that have ** been written to the journal have actually reached the surface of the ** disk and can be restored in the event of a hot-journal rollback. ** -** If the Pager.needSync flag is not set, then this function is a -** no-op. Otherwise, the actions required depend on the journal-mode -** and the device characteristics of the the file-system, as follows: +** If the Pager.noSync flag is set, then this function is a no-op. +** Otherwise, the actions required depend on the journal-mode and the +** device characteristics of the the file-system, as follows: ** ** * If the journal file is an in-memory journal file, no action need ** be taken. ** ** * Otherwise, if the device does not support the SAFE_APPEND property, @@ -37232,22 +38315,29 @@ ** <update nRec field> ** } ** if( NOT SEQUENTIAL ) xSync(<journal file>); ** } ** -** The Pager.needSync flag is never be set for temporary files, or any -** file operating in no-sync mode (Pager.noSync set to non-zero). -** ** If successful, this routine clears the PGHDR_NEED_SYNC flag of every ** page currently held in memory before returning SQLITE_OK. If an IO ** error is encountered, then the IO error code is returned to the caller. */ -static int syncJournal(Pager *pPager){ - if( pPager->needSync ){ +static int syncJournal(Pager *pPager, int newHdr){ + int rc; /* Return code */ + + assert( pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + ); + assert( assert_pager_state(pPager) ); + assert( !pagerUseWal(pPager) ); + + rc = sqlite3PagerExclusiveLock(pPager); + if( rc!=SQLITE_OK ) return rc; + + if( !pPager->noSync ){ assert( !pPager->tempFile ); - if( pPager->journalMode!=PAGER_JOURNALMODE_MEMORY ){ - int rc; /* Return code */ + if( isOpen(pPager->jfd) && pPager->journalMode!=PAGER_JOURNALMODE_MEMORY ){ const int iDc = sqlite3OsDeviceCharacteristics(pPager->fd); assert( isOpen(pPager->jfd) ); if( 0==(iDc&SQLITE_IOCAP_SAFE_APPEND) ){ /* This block deals with an obscure problem. If the last connection @@ -37318,21 +38408,29 @@ rc = sqlite3OsSync(pPager->jfd, pPager->sync_flags| (pPager->sync_flags==SQLITE_SYNC_FULL?SQLITE_SYNC_DATAONLY:0) ); if( rc!=SQLITE_OK ) return rc; } - } - - /* The journal file was just successfully synced. Set Pager.needSync - ** to zero and clear the PGHDR_NEED_SYNC flag on all pagess. - */ - pPager->needSync = 0; - pPager->journalStarted = 1; - pPager->journalHdr = pPager->journalOff; - sqlite3PcacheClearSyncFlags(pPager->pPCache); + + pPager->journalHdr = pPager->journalOff; + if( newHdr && 0==(iDc&SQLITE_IOCAP_SAFE_APPEND) ){ + pPager->nRec = 0; + rc = writeJournalHdr(pPager); + if( rc!=SQLITE_OK ) return rc; + } + }else{ + pPager->journalHdr = pPager->journalOff; + } } + /* Unless the pager is in noSync mode, the journal file was just + ** successfully synced. Either way, clear the PGHDR_NEED_SYNC flag on + ** all pages. + */ + sqlite3PcacheClearSyncFlags(pPager->pPCache); + pPager->eState = PAGER_WRITER_DBMOD; + assert( assert_pager_state(pPager) ); return SQLITE_OK; } /* ** The argument is the first in a linked list of dirty pages connected @@ -37365,31 +38463,16 @@ ** If everything is successful, SQLITE_OK is returned. If an IO error ** occurs, an IO error code is returned. Or, if the EXCLUSIVE lock cannot ** be obtained, SQLITE_BUSY is returned. */ static int pager_write_pagelist(Pager *pPager, PgHdr *pList){ - int rc; /* Return code */ - - /* At this point there may be either a RESERVED or EXCLUSIVE lock on the - ** database file. If there is already an EXCLUSIVE lock, the following - ** call is a no-op. - ** - ** Moving the lock from RESERVED to EXCLUSIVE actually involves going - ** through an intermediate state PENDING. A PENDING lock prevents new - ** readers from attaching to the database but is unsufficient for us to - ** write. The idea of a PENDING lock is to prevent new readers from - ** coming in while we wait for existing readers to clear. - ** - ** While the pager is in the RESERVED state, the original database file - ** is unchanged and we can rollback without having to playback the - ** journal into the original database file. Once we transition to - ** EXCLUSIVE, it means the database file has been changed and any rollback - ** will require a journal playback. - */ + int rc = SQLITE_OK; /* Return code */ + + /* This function is only called for rollback pagers in WRITER_DBMOD state. */ assert( !pagerUseWal(pPager) ); - assert( pPager->state>=PAGER_RESERVED ); - rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); + assert( pPager->eState==PAGER_WRITER_DBMOD ); + assert( pPager->eLock==EXCLUSIVE_LOCK ); /* If the file is a temp-file has not yet been opened, open it now. It ** is not possible for rc to be other than SQLITE_OK if this branch ** is taken, as pager_wait_on_lock() is a no-op for temp-files. */ @@ -37400,13 +38483,14 @@ /* Before the first write, give the VFS a hint of what the final ** file size will be. */ assert( rc!=SQLITE_OK || isOpen(pPager->fd) ); - if( rc==SQLITE_OK && pPager->dbSize>(pPager->dbOrigSize+1) ){ + if( rc==SQLITE_OK && pPager->dbSize>pPager->dbHintSize ){ sqlite3_int64 szFile = pPager->pageSize * (sqlite3_int64)pPager->dbSize; sqlite3OsFileControl(pPager->fd, SQLITE_FCNTL_SIZE_HINT, &szFile); + pPager->dbHintSize = pPager->dbSize; } while( rc==SQLITE_OK && pList ){ Pgno pgno = pList->pgno; @@ -37419,10 +38503,12 @@ ** set (set by sqlite3PagerDontWrite()). */ if( pgno<=pPager->dbSize && 0==(pList->flags&PGHDR_DONT_WRITE) ){ i64 offset = (pgno-1)*(i64)pPager->pageSize; /* Offset to write */ char *pData; /* Data to write */ + + assert( (pList->flags&PGHDR_NEED_SYNC)==0 ); /* Encode the database */ CODEC2(pPager, pList->pData, pgno, 6, return SQLITE_NOMEM, pData); /* Write out the page data. */ @@ -37448,13 +38534,11 @@ PAGER_INCR(sqlite3_pager_writedb_count); PAGER_INCR(pPager->nWrite); }else{ PAGERTRACE(("NOSTORE %d page %d\n", PAGERID(pPager), pgno)); } -#ifdef SQLITE_CHECK_PAGES - pList->pageHash = pager_pagehash(pList); -#endif + pager_set_pagehash(pList); pList = pList->pDirty; } return rc; } @@ -37562,13 +38646,18 @@ ** pages belonging to the same sector. ** ** The doNotSpill flag inhibits all cache spilling regardless of whether ** or not a sync is required. This is set during a rollback. ** - ** Spilling is also inhibited when in an error state. + ** Spilling is also prohibited when in an error state since that could + ** lead to database corruption. In the current implementaton it + ** is impossible for sqlite3PCacheFetch() to be called with createFlag==1 + ** while in the error state, hence it is impossible for this routine to + ** be called in the error state. Nevertheless, we include a NEVER() + ** test for the error state as a safeguard against future changes. */ - if( pPager->errCode ) return SQLITE_OK; + if( NEVER(pPager->errCode) ) return SQLITE_OK; if( pPager->doNotSpill ) return SQLITE_OK; if( pPager->doNotSyncSpill && (pPg->flags & PGHDR_NEED_SYNC)!=0 ){ return SQLITE_OK; } @@ -37582,20 +38671,14 @@ rc = pagerWalFrames(pPager, pPg, 0, 0, 0); } }else{ /* Sync the journal file if required. */ - if( pPg->flags&PGHDR_NEED_SYNC ){ - assert( !pPager->noSync ); - rc = syncJournal(pPager); - if( rc==SQLITE_OK && - !(pPager->journalMode==PAGER_JOURNALMODE_MEMORY) && - !(sqlite3OsDeviceCharacteristics(pPager->fd)&SQLITE_IOCAP_SAFE_APPEND) - ){ - pPager->nRec = 0; - rc = writeJournalHdr(pPager); - } + if( pPg->flags&PGHDR_NEED_SYNC + || pPager->eState==PAGER_WRITER_CACHEMOD + ){ + rc = syncJournal(pPager, 1); } /* If the page number of this page is larger than the current size of ** the database image, it may need to be written to the sub-journal. ** This is because the call to pager_write_pagelist() below will not @@ -37629,10 +38712,11 @@ rc = subjournalPage(pPg); } /* Write the contents of the page out to the database file. */ if( rc==SQLITE_OK ){ + assert( (pPg->flags&PGHDR_NEED_SYNC)==0 ); rc = pager_write_pagelist(pPager, pPg); } } /* Mark the page as clean. */ @@ -37639,11 +38723,11 @@ if( rc==SQLITE_OK ){ PAGERTRACE(("STRESS %d page %d\n", PAGERID(pPager), pPg->pgno)); sqlite3PcacheMakeClean(pPg); } - return pager_error(pPager, rc); + return pager_error(pPager, rc); } /* ** Allocate and initialize a new Pager object and put a pointer to it @@ -37694,11 +38778,11 @@ char *zPathname = 0; /* Full path to database file */ int nPathname = 0; /* Number of bytes in zPathname */ int useJournal = (flags & PAGER_OMIT_JOURNAL)==0; /* False to omit journal */ int noReadlock = (flags & PAGER_NO_READLOCK)!=0; /* True to omit read-lock */ int pcacheSize = sqlite3PcacheSize(); /* Bytes to allocate for PCache */ - u16 szPageDflt = SQLITE_DEFAULT_PAGE_SIZE; /* Default page size */ + u32 szPageDflt = SQLITE_DEFAULT_PAGE_SIZE; /* Default page size */ /* Figure out how much space is required for each journal file-handle ** (there are two of them, the main journal and the sub-journal). This ** is the maximum space required for an in-memory journal file handle ** and a regular journal file-handle. Note that a "regular journal-handle" @@ -37712,10 +38796,17 @@ journalFileSize = ROUND8(sqlite3MemJournalSize()); } /* Set the output variable to NULL in case an error occurs. */ *ppPager = 0; + +#ifndef SQLITE_OMIT_MEMORYDB + if( flags & PAGER_MEMORY ){ + memDb = 1; + zFilename = 0; + } +#endif /* Compute and store the full pathname in an allocated buffer pointed ** to by zPathname, length nPathname. Or, if this is a temporary file, ** leave both nPathname and zPathname set to 0. */ @@ -37723,21 +38814,12 @@ nPathname = pVfs->mxPathname+1; zPathname = sqlite3Malloc(nPathname*2); if( zPathname==0 ){ return SQLITE_NOMEM; } -#ifndef SQLITE_OMIT_MEMORYDB - if( strcmp(zFilename,":memory:")==0 ){ - memDb = 1; - zPathname[0] = 0; - }else -#endif - { - zPathname[0] = 0; /* Make sure initialized even if FullPathname() fails */ - rc = sqlite3OsFullPathname(pVfs, zFilename, nPathname, zPathname); - } - + zPathname[0] = 0; /* Make sure initialized even if FullPathname() fails */ + rc = sqlite3OsFullPathname(pVfs, zFilename, nPathname, zPathname); nPathname = sqlite3Strlen30(zPathname); if( rc==SQLITE_OK && nPathname+8>pVfs->mxPathname ){ /* This branch is taken when the journal path required by ** the database being opened will be more than pVfs->mxPathname ** bytes in length. This means the database cannot be opened, @@ -37788,34 +38870,31 @@ pPager->zFilename = (char*)(pPtr += journalFileSize); assert( EIGHT_BYTE_ALIGNMENT(pPager->jfd) ); /* Fill in the Pager.zFilename and Pager.zJournal buffers, if required. */ if( zPathname ){ + assert( nPathname>0 ); pPager->zJournal = (char*)(pPtr += nPathname + 1); memcpy(pPager->zFilename, zPathname, nPathname); memcpy(pPager->zJournal, zPathname, nPathname); memcpy(&pPager->zJournal[nPathname], "-journal", 8); - if( pPager->zFilename[0]==0 ){ - pPager->zJournal[0] = 0; - } #ifndef SQLITE_OMIT_WAL - else{ - pPager->zWal = &pPager->zJournal[nPathname+8+1]; - memcpy(pPager->zWal, zPathname, nPathname); - memcpy(&pPager->zWal[nPathname], "-wal", 4); - } + pPager->zWal = &pPager->zJournal[nPathname+8+1]; + memcpy(pPager->zWal, zPathname, nPathname); + memcpy(&pPager->zWal[nPathname], "-wal", 4); #endif sqlite3_free(zPathname); } pPager->pVfs = pVfs; pPager->vfsFlags = vfsFlags; /* Open the pager file. */ - if( zFilename && zFilename[0] && !memDb ){ + if( zFilename && zFilename[0] ){ int fout = 0; /* VFS flags returned by xOpen() */ rc = sqlite3OsOpen(pVfs, pPager->zFilename, pPager->fd, vfsFlags, &fout); + assert( !memDb ); readOnly = (fout&SQLITE_OPEN_READONLY); /* If the file was successfully opened for read/write access, ** choose a default page size in case we have to create the ** database file. The default page size is the maximum of: @@ -37829,11 +38908,11 @@ assert(SQLITE_DEFAULT_PAGE_SIZE<=SQLITE_MAX_DEFAULT_PAGE_SIZE); if( szPageDflt<pPager->sectorSize ){ if( pPager->sectorSize>SQLITE_MAX_DEFAULT_PAGE_SIZE ){ szPageDflt = SQLITE_MAX_DEFAULT_PAGE_SIZE; }else{ - szPageDflt = (u16)pPager->sectorSize; + szPageDflt = (u32)pPager->sectorSize; } } #ifdef SQLITE_ENABLE_ATOMIC_WRITE { int iDc = sqlite3OsDeviceCharacteristics(pPager->fd); @@ -37857,11 +38936,12 @@ ** This branch is also run for an in-memory database. An in-memory ** database is the same as a temp-file that is never written out to ** disk and uses an in-memory rollback journal. */ tempFile = 1; - pPager->state = PAGER_EXCLUSIVE; + pPager->eState = PAGER_READER; + pPager->eLock = EXCLUSIVE_LOCK; readOnly = (vfsFlags&SQLITE_OPEN_READONLY); } /* The following call to PagerSetPagesize() serves to set the value of ** Pager.pageSize and to allocate the Pager.pTmpSpace buffer. @@ -37894,27 +38974,27 @@ pPager->useJournal = (u8)useJournal; pPager->noReadlock = (noReadlock && readOnly) ?1:0; /* pPager->stmtOpen = 0; */ /* pPager->stmtInUse = 0; */ /* pPager->nRef = 0; */ - pPager->dbSizeValid = (u8)memDb; /* pPager->stmtSize = 0; */ /* pPager->stmtJSize = 0; */ /* pPager->nPage = 0; */ pPager->mxPgno = SQLITE_MAX_PAGE_COUNT; /* pPager->state = PAGER_UNLOCK; */ +#if 0 assert( pPager->state == (tempFile ? PAGER_EXCLUSIVE : PAGER_UNLOCK) ); +#endif /* pPager->errMask = 0; */ pPager->tempFile = (u8)tempFile; assert( tempFile==PAGER_LOCKINGMODE_NORMAL || tempFile==PAGER_LOCKINGMODE_EXCLUSIVE ); assert( PAGER_LOCKINGMODE_EXCLUSIVE==1 ); pPager->exclusiveMode = (u8)tempFile; pPager->changeCountDone = pPager->tempFile; pPager->memDb = (u8)memDb; pPager->readOnly = (u8)readOnly; - /* pPager->needSync = 0; */ assert( useJournal || pPager->tempFile ); pPager->noSync = pPager->tempFile; pPager->fullSync = pPager->noSync ?0:1; pPager->sync_flags = SQLITE_SYNC_NORMAL; /* pPager->pFirst = 0; */ @@ -37975,24 +39055,24 @@ sqlite3_vfs * const pVfs = pPager->pVfs; int rc = SQLITE_OK; /* Return code */ int exists = 1; /* True if a journal file is present */ int jrnlOpen = !!isOpen(pPager->jfd); - assert( pPager!=0 ); assert( pPager->useJournal ); assert( isOpen(pPager->fd) ); - assert( pPager->state <= PAGER_SHARED ); + assert( pPager->eState==PAGER_OPEN ); + assert( jrnlOpen==0 || ( sqlite3OsDeviceCharacteristics(pPager->jfd) & SQLITE_IOCAP_UNDELETABLE_WHEN_OPEN )); *pExists = 0; if( !jrnlOpen ){ rc = sqlite3OsAccess(pVfs, pPager->zJournal, SQLITE_ACCESS_EXISTS, &exists); } if( rc==SQLITE_OK && exists ){ - int locked; /* True if some process holds a RESERVED lock */ + int locked = 0; /* True if some process holds a RESERVED lock */ /* Race condition here: Another process might have been holding the ** the RESERVED lock and have a journal open at the sqlite3OsAccess() ** call above, but then delete the journal and drop the lock before ** we get to the following sqlite3OsCheckReservedLock() call. If that @@ -38000,25 +39080,25 @@ ** in fact there is none. This results in a false-positive which will ** be dealt with by the playback routine. Ticket #3883. */ rc = sqlite3OsCheckReservedLock(pPager->fd, &locked); if( rc==SQLITE_OK && !locked ){ - int nPage; + Pgno nPage; /* Number of pages in database file */ /* Check the size of the database file. If it consists of 0 pages, ** then delete the journal file. See the header comment above for ** the reasoning here. Delete the obsolete journal file under ** a RESERVED lock to avoid race conditions and to avoid violating ** [H33020]. */ - rc = sqlite3PagerPagecount(pPager, &nPage); + rc = pagerPagecount(pPager, &nPage); if( rc==SQLITE_OK ){ if( nPage==0 ){ sqlite3BeginBenignMalloc(); - if( sqlite3OsLock(pPager->fd, RESERVED_LOCK)==SQLITE_OK ){ + if( pagerLockDb(pPager, RESERVED_LOCK)==SQLITE_OK ){ sqlite3OsDelete(pVfs, pPager->zJournal, 0); - sqlite3OsUnlock(pPager->fd, SHARED_LOCK); + pagerUnlockDb(pPager, SHARED_LOCK); } sqlite3EndBenignMalloc(); }else{ /* The journal file exists and no other connection has a reserved ** or greater lock on the database file. Now check that there is @@ -38067,11 +39147,11 @@ ** has been successfully called. If a shared-lock is already held when ** this function is called, it is a no-op. ** ** The following operations are also performed by this function. ** -** 1) If the pager is currently in PAGER_UNLOCK state (no lock held +** 1) If the pager is currently in PAGER_OPEN state (no lock held ** on the database file), then an attempt is made to obtain a ** SHARED lock on the database file. Immediately after obtaining ** the SHARED lock, the file-system is checked for a hot-journal, ** which is played back if present. Following any hot-journal ** rollback, the contents of the cache are validated by checking @@ -38082,70 +39162,51 @@ ** no outstanding references to any pages, and is in the error state, ** then an attempt is made to clear the error state by discarding ** the contents of the page cache and rolling back any open journal ** file. ** -** If the operation described by (2) above is not attempted, and if the -** pager is in an error state other than SQLITE_FULL when this is called, -** the error state error code is returned. It is permitted to read the -** database when in SQLITE_FULL error state. -** -** Otherwise, if everything is successful, SQLITE_OK is returned. If an -** IO error occurs while locking the database, checking for a hot-journal -** file or rolling back a journal file, the IO error code is returned. +** If everything is successful, SQLITE_OK is returned. If an IO error +** occurs while locking the database, checking for a hot-journal file or +** rolling back a journal file, the IO error code is returned. */ SQLITE_PRIVATE int sqlite3PagerSharedLock(Pager *pPager){ int rc = SQLITE_OK; /* Return code */ - int isErrorReset = 0; /* True if recovering from error state */ /* This routine is only called from b-tree and only when there are no - ** outstanding pages */ + ** outstanding pages. This implies that the pager state should either + ** be OPEN or READER. READER is only possible if the pager is or was in + ** exclusive access mode. + */ assert( sqlite3PcacheRefCount(pPager->pPCache)==0 ); + assert( assert_pager_state(pPager) ); + assert( pPager->eState==PAGER_OPEN || pPager->eState==PAGER_READER ); if( NEVER(MEMDB && pPager->errCode) ){ return pPager->errCode; } - /* If this database is in an error-state, now is a chance to clear - ** the error. Discard the contents of the pager-cache and rollback - ** any hot journal in the file-system. - */ - if( pPager->errCode ){ - if( isOpen(pPager->jfd) || pPager->zJournal ){ - isErrorReset = 1; - } - pPager->errCode = SQLITE_OK; - pager_reset(pPager); - } - - if( pagerUseWal(pPager) ){ - rc = pagerBeginReadTransaction(pPager); - }else if( pPager->state==PAGER_UNLOCK || isErrorReset ){ - sqlite3_vfs * const pVfs = pPager->pVfs; - int isHotJournal = 0; + if( !pagerUseWal(pPager) && pPager->eState==PAGER_OPEN ){ + int bHotJournal = 1; /* True if there exists a hot journal-file */ + assert( !MEMDB ); - assert( sqlite3PcacheRefCount(pPager->pPCache)==0 ); - if( pPager->noReadlock ){ - assert( pPager->readOnly ); - pPager->state = PAGER_SHARED; - }else{ + assert( pPager->noReadlock==0 || pPager->readOnly ); + + if( pPager->noReadlock==0 ){ rc = pager_wait_on_lock(pPager, SHARED_LOCK); if( rc!=SQLITE_OK ){ - assert( pPager->state==PAGER_UNLOCK ); - return pager_error(pPager, rc); + assert( pPager->eLock==NO_LOCK || pPager->eLock==UNKNOWN_LOCK ); + goto failed; } } - assert( pPager->state>=SHARED_LOCK ); /* If a journal file exists, and there is no RESERVED lock on the ** database file, then it either needs to be played back or deleted. */ - if( !isErrorReset ){ - assert( pPager->state <= PAGER_SHARED ); - rc = hasHotJournal(pPager, &isHotJournal); - if( rc!=SQLITE_OK ){ - goto failed; - } - } - if( isErrorReset || isHotJournal ){ + if( pPager->eLock<=SHARED_LOCK ){ + rc = hasHotJournal(pPager, &bHotJournal); + } + if( rc!=SQLITE_OK ){ + goto failed; + } + if( bHotJournal ){ /* Get an EXCLUSIVE lock on the database file. At this point it is ** important that a RESERVED lock is not obtained on the way to the ** EXCLUSIVE lock. If it were, another process might open the ** database file, detect the RESERVED lock, and conclude that the ** database is safe to read while this process is still rolling the @@ -38153,62 +39214,49 @@ ** ** Because the intermediate RESERVED lock is not requested, any ** other process attempting to access the database file will get to ** this point in the code and fail to obtain its own EXCLUSIVE lock ** on the database file. - */ - if( pPager->state<EXCLUSIVE_LOCK ){ - rc = sqlite3OsLock(pPager->fd, EXCLUSIVE_LOCK); - if( rc!=SQLITE_OK ){ - rc = pager_error(pPager, rc); - goto failed; - } - pPager->state = PAGER_EXCLUSIVE; - } - - /* Open the journal for read/write access. This is because in - ** exclusive-access mode the file descriptor will be kept open and - ** possibly used for a transaction later on. On some systems, the - ** OsTruncate() call used in exclusive-access mode also requires - ** a read/write file handle. - */ - if( !isOpen(pPager->jfd) ){ - int res; - rc = sqlite3OsAccess(pVfs,pPager->zJournal,SQLITE_ACCESS_EXISTS,&res); - if( rc==SQLITE_OK ){ - if( res ){ - int fout = 0; - int f = SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_JOURNAL; - assert( !pPager->tempFile ); - rc = sqlite3OsOpen(pVfs, pPager->zJournal, pPager->jfd, f, &fout); - assert( rc!=SQLITE_OK || isOpen(pPager->jfd) ); - if( rc==SQLITE_OK && fout&SQLITE_OPEN_READONLY ){ - rc = SQLITE_CANTOPEN_BKPT; - sqlite3OsClose(pPager->jfd); - } - }else{ - /* If the journal does not exist, it usually means that some - ** other connection managed to get in and roll it back before - ** this connection obtained the exclusive lock above. Or, it - ** may mean that the pager was in the error-state when this - ** function was called and the journal file does not exist. */ - rc = pager_end_transaction(pPager, 0); - } - } - } - if( rc!=SQLITE_OK ){ - goto failed; - } - - /* Reset the journal status fields to indicates that we have no - ** rollback journal at this time. */ - pPager->journalStarted = 0; - pPager->journalOff = 0; - pPager->setMaster = 0; - pPager->journalHdr = 0; - - /* Make sure the journal file has been synced to disk. */ + ** + ** Unless the pager is in locking_mode=exclusive mode, the lock is + ** downgraded to SHARED_LOCK before this function returns. + */ + rc = pagerLockDb(pPager, EXCLUSIVE_LOCK); + if( rc!=SQLITE_OK ){ + goto failed; + } + + /* If it is not already open and the file exists on disk, open the + ** journal for read/write access. Write access is required because + ** in exclusive-access mode the file descriptor will be kept open + ** and possibly used for a transaction later on. Also, write-access + ** is usually required to finalize the journal in journal_mode=persist + ** mode (and also for journal_mode=truncate on some systems). + ** + ** If the journal does not exist, it usually means that some + ** other connection managed to get in and roll it back before + ** this connection obtained the exclusive lock above. Or, it + ** may mean that the pager was in the error-state when this + ** function was called and the journal file does not exist. + */ + if( !isOpen(pPager->jfd) ){ + sqlite3_vfs * const pVfs = pPager->pVfs; + int bExists; /* True if journal file exists */ + rc = sqlite3OsAccess( + pVfs, pPager->zJournal, SQLITE_ACCESS_EXISTS, &bExists); + if( rc==SQLITE_OK && bExists ){ + int fout = 0; + int f = SQLITE_OPEN_READWRITE|SQLITE_OPEN_MAIN_JOURNAL; + assert( !pPager->tempFile ); + rc = sqlite3OsOpen(pVfs, pPager->zJournal, pPager->jfd, f, &fout); + assert( rc!=SQLITE_OK || isOpen(pPager->jfd) ); + if( rc==SQLITE_OK && fout&SQLITE_OPEN_READONLY ){ + rc = SQLITE_CANTOPEN_BKPT; + sqlite3OsClose(pPager->jfd); + } + } + } /* Playback and delete the journal. Drop the database write ** lock and reacquire the read lock. Purge the cache before ** playing back the hot-journal so that we don't end up with ** an inconsistent cache. Sync the hot journal before playing @@ -38215,25 +39263,50 @@ ** it back since the process that crashed and left the hot journal ** probably did not sync it and we are required to always sync ** the journal before playing it back. */ if( isOpen(pPager->jfd) ){ + assert( rc==SQLITE_OK ); rc = pagerSyncHotJournal(pPager); if( rc==SQLITE_OK ){ rc = pager_playback(pPager, 1); - } - if( rc!=SQLITE_OK ){ - rc = pager_error(pPager, rc); - goto failed; - } - } - assert( (pPager->state==PAGER_SHARED) - || (pPager->exclusiveMode && pPager->state>PAGER_SHARED) + pPager->eState = PAGER_OPEN; + } + }else if( !pPager->exclusiveMode ){ + pagerUnlockDb(pPager, SHARED_LOCK); + } + + if( rc!=SQLITE_OK ){ + /* This branch is taken if an error occurs while trying to open + ** or roll back a hot-journal while holding an EXCLUSIVE lock. The + ** pager_unlock() routine will be called before returning to unlock + ** the file. If the unlock attempt fails, then Pager.eLock must be + ** set to UNKNOWN_LOCK (see the comment above the #define for + ** UNKNOWN_LOCK above for an explanation). + ** + ** In order to get pager_unlock() to do this, set Pager.eState to + ** PAGER_ERROR now. This is not actually counted as a transition + ** to ERROR state in the state diagram at the top of this file, + ** since we know that the same call to pager_unlock() will very + ** shortly transition the pager object to the OPEN state. Calling + ** assert_pager_state() would fail now, as it should not be possible + ** to be in ERROR state when there are zero outstanding page + ** references. + */ + pager_error(pPager, rc); + goto failed; + } + + assert( pPager->eState==PAGER_OPEN ); + assert( (pPager->eLock==SHARED_LOCK) + || (pPager->exclusiveMode && pPager->eLock>SHARED_LOCK) ); } - if( pPager->pBackup || sqlite3PcachePagecount(pPager->pPCache)>0 ){ + if( !pPager->tempFile + && (pPager->pBackup || sqlite3PcachePagecount(pPager->pPCache)>0) + ){ /* The shared-lock has just been acquired on the database file ** and there are already pages in the cache (from a previous ** read or write transaction). Check to see if the database ** has been modified. If the database has changed, flush the ** cache. @@ -38246,18 +39319,15 @@ ** ** There is a vanishingly small chance that a change will not be ** detected. The chance of an undetected change is so small that ** it can be neglected. */ - int nPage = 0; + Pgno nPage = 0; char dbFileVers[sizeof(pPager->dbFileVers)]; - sqlite3PagerPagecount(pPager, &nPage); - if( pPager->errCode ){ - rc = pPager->errCode; - goto failed; - } + rc = pagerPagecount(pPager, &nPage); + if( rc ) goto failed; if( nPage>0 ){ IOTRACE(("CKVERS %p %d\n", pPager, sizeof(dbFileVers))); rc = sqlite3OsRead(pPager->fd, &dbFileVers, sizeof(dbFileVers), 24); if( rc!=SQLITE_OK ){ @@ -38269,22 +39339,36 @@ if( memcmp(pPager->dbFileVers, dbFileVers, sizeof(dbFileVers))!=0 ){ pager_reset(pPager); } } - assert( pPager->exclusiveMode || pPager->state==PAGER_SHARED ); /* If there is a WAL file in the file-system, open this database in WAL ** mode. Otherwise, the following function call is a no-op. */ rc = pagerOpenWalIfPresent(pPager); +#ifndef SQLITE_OMIT_WAL + assert( pPager->pWal==0 || rc==SQLITE_OK ); +#endif + } + + if( pagerUseWal(pPager) ){ + assert( rc==SQLITE_OK ); + rc = pagerBeginReadTransaction(pPager); + } + + if( pPager->eState==PAGER_OPEN && rc==SQLITE_OK ){ + rc = pagerPagecount(pPager, &pPager->dbSize); } failed: if( rc!=SQLITE_OK ){ - /* pager_unlock() is a no-op for exclusive mode and in-memory databases. */ + assert( !MEMDB ); pager_unlock(pPager); + assert( pPager->eState==PAGER_OPEN ); + }else{ + pPager->eState = PAGER_READER; } return rc; } /* @@ -38294,13 +39378,11 @@ ** Except, in locking_mode=EXCLUSIVE when there is nothing to in ** the rollback journal, the unlock is not performed and there is ** nothing to rollback, so this routine is a no-op. */ static void pagerUnlockIfUnused(Pager *pPager){ - if( (sqlite3PcacheRefCount(pPager->pPCache)==0) - && (!pPager->exclusiveMode || pPager->journalOff>0) - ){ + if( (sqlite3PcacheRefCount(pPager->pPCache)==0) ){ pagerUnlockAndRollback(pPager); } } /* @@ -38360,20 +39442,20 @@ int noContent /* Do not bother reading content from disk if true */ ){ int rc; PgHdr *pPg; + assert( pPager->eState>=PAGER_READER ); assert( assert_pager_state(pPager) ); - assert( pPager->state>PAGER_UNLOCK ); if( pgno==0 ){ return SQLITE_CORRUPT_BKPT; } /* If the pager is in the error state, return an error immediately. ** Otherwise, request the page from the PCache layer. */ - if( pPager->errCode!=SQLITE_OK && pPager->errCode!=SQLITE_FULL ){ + if( pPager->errCode!=SQLITE_OK ){ rc = pPager->errCode; }else{ rc = sqlite3PcacheFetch(pPager->pPCache, pgno, 1, ppPage); } @@ -38395,11 +39477,10 @@ return SQLITE_OK; }else{ /* The pager cache has created a new page. Its content needs to ** be initialized. */ - int nMax; PAGER_INCR(pPager->nMiss); pPg = *ppPage; pPg->pPager = pPager; @@ -38408,16 +39489,11 @@ if( pgno>PAGER_MAX_PGNO || pgno==PAGER_MJ_PGNO(pPager) ){ rc = SQLITE_CORRUPT_BKPT; goto pager_acquire_err; } - rc = sqlite3PagerPagecount(pPager, &nMax); - if( rc!=SQLITE_OK ){ - goto pager_acquire_err; - } - - if( MEMDB || nMax<(int)pgno || noContent || !isOpen(pPager->fd) ){ + if( MEMDB || pPager->dbSize<pgno || noContent || !isOpen(pPager->fd) ){ if( pgno>pPager->mxPgno ){ rc = SQLITE_FULL; goto pager_acquire_err; } if( noContent ){ @@ -38443,13 +39519,11 @@ rc = readDbPage(pPg); if( rc!=SQLITE_OK ){ goto pager_acquire_err; } } -#ifdef SQLITE_CHECK_PAGES - pPg->pageHash = pager_pagehash(pPg); -#endif + pager_set_pagehash(pPg); } return SQLITE_OK; pager_acquire_err: @@ -38464,13 +39538,11 @@ } /* ** Acquire a page if it is already in the in-memory cache. Do ** not read the page from disk. Return a pointer to the page, -** or 0 if the page is not in cache. Also, return 0 if the -** pager is in PAGER_UNLOCK state when this function is called, -** or if the pager is in an error state other than SQLITE_FULL. +** or 0 if the page is not in cache. ** ** See also sqlite3PagerGet(). The difference between this routine ** and sqlite3PagerGet() is that _get() will go to the disk and read ** in the page if the page is not already in cache. This routine ** returns NULL if the page is not in cache or if a disk I/O error @@ -38479,11 +39551,11 @@ SQLITE_PRIVATE DbPage *sqlite3PagerLookup(Pager *pPager, Pgno pgno){ PgHdr *pPg = 0; assert( pPager!=0 ); assert( pgno!=0 ); assert( pPager->pPCache!=0 ); - assert( pPager->state > PAGER_UNLOCK ); + assert( pPager->eState>=PAGER_READER && pPager->eState!=PAGER_ERROR ); sqlite3PcacheFetch(pPager->pPCache, pgno, 0, &pPg); return pPg; } /* @@ -38524,73 +39596,71 @@ ** SQLITE_NOMEM if the attempt to allocate Pager.pInJournal fails, or ** an IO error code if opening or writing the journal file fails. */ static int pager_open_journal(Pager *pPager){ int rc = SQLITE_OK; /* Return code */ - int nPage; /* Size of database file */ sqlite3_vfs * const pVfs = pPager->pVfs; /* Local cache of vfs pointer */ - assert( pPager->state>=PAGER_RESERVED ); - assert( pPager->useJournal ); - assert( pPager->journalMode!=PAGER_JOURNALMODE_OFF ); + assert( pPager->eState==PAGER_WRITER_LOCKED ); + assert( assert_pager_state(pPager) ); assert( pPager->pInJournal==0 ); /* If already in the error state, this function is a no-op. But on ** the other hand, this routine is never called if we are already in ** an error state. */ if( NEVER(pPager->errCode) ) return pPager->errCode; - testcase( pPager->dbSizeValid==0 ); - rc = sqlite3PagerPagecount(pPager, &nPage); - if( rc ) return rc; - pPager->pInJournal = sqlite3BitvecCreate(nPage); - if( pPager->pInJournal==0 ){ - return SQLITE_NOMEM; - } - - /* Open the journal file if it is not already open. */ - if( !isOpen(pPager->jfd) ){ - if( pPager->journalMode==PAGER_JOURNALMODE_MEMORY ){ - sqlite3MemJournalOpen(pPager->jfd); - }else{ - const int flags = /* VFS flags to open journal file */ - SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE| - (pPager->tempFile ? - (SQLITE_OPEN_DELETEONCLOSE|SQLITE_OPEN_TEMP_JOURNAL): - (SQLITE_OPEN_MAIN_JOURNAL) - ); -#ifdef SQLITE_ENABLE_ATOMIC_WRITE - rc = sqlite3JournalOpen( - pVfs, pPager->zJournal, pPager->jfd, flags, jrnlBufferSize(pPager) - ); -#else - rc = sqlite3OsOpen(pVfs, pPager->zJournal, pPager->jfd, flags, 0); -#endif - } - assert( rc!=SQLITE_OK || isOpen(pPager->jfd) ); - } - - - /* Write the first journal header to the journal file and open - ** the sub-journal if necessary. - */ - if( rc==SQLITE_OK ){ - /* TODO: Check if all of these are really required. */ - pPager->dbOrigSize = pPager->dbSize; - pPager->journalStarted = 0; - pPager->needSync = 0; - pPager->nRec = 0; - pPager->journalOff = 0; - pPager->setMaster = 0; - pPager->journalHdr = 0; - rc = writeJournalHdr(pPager); + if( !pagerUseWal(pPager) && pPager->journalMode!=PAGER_JOURNALMODE_OFF ){ + pPager->pInJournal = sqlite3BitvecCreate(pPager->dbSize); + if( pPager->pInJournal==0 ){ + return SQLITE_NOMEM; + } + + /* Open the journal file if it is not already open. */ + if( !isOpen(pPager->jfd) ){ + if( pPager->journalMode==PAGER_JOURNALMODE_MEMORY ){ + sqlite3MemJournalOpen(pPager->jfd); + }else{ + const int flags = /* VFS flags to open journal file */ + SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE| + (pPager->tempFile ? + (SQLITE_OPEN_DELETEONCLOSE|SQLITE_OPEN_TEMP_JOURNAL): + (SQLITE_OPEN_MAIN_JOURNAL) + ); + #ifdef SQLITE_ENABLE_ATOMIC_WRITE + rc = sqlite3JournalOpen( + pVfs, pPager->zJournal, pPager->jfd, flags, jrnlBufferSize(pPager) + ); + #else + rc = sqlite3OsOpen(pVfs, pPager->zJournal, pPager->jfd, flags, 0); + #endif + } + assert( rc!=SQLITE_OK || isOpen(pPager->jfd) ); + } + + + /* Write the first journal header to the journal file and open + ** the sub-journal if necessary. + */ + if( rc==SQLITE_OK ){ + /* TODO: Check if all of these are really required. */ + pPager->nRec = 0; + pPager->journalOff = 0; + pPager->setMaster = 0; + pPager->journalHdr = 0; + rc = writeJournalHdr(pPager); + } } if( rc!=SQLITE_OK ){ sqlite3BitvecDestroy(pPager->pInJournal); pPager->pInJournal = 0; + }else{ + assert( pPager->eState==PAGER_WRITER_LOCKED ); + pPager->eState = PAGER_WRITER_CACHEMOD; } + return rc; } /* ** Begin a write-transaction on the specified pager object. If a @@ -38599,18 +39669,10 @@ ** If the exFlag argument is false, then acquire at least a RESERVED ** lock on the database file. If exFlag is true, then acquire at least ** an EXCLUSIVE lock. If such a lock is already held, no locking ** functions need be called. ** -** If this is not a temporary or in-memory file and, the journal file is -** opened if it has not been already. For a temporary file, the opening -** of the journal file is deferred until there is an actual need to -** write to the journal. TODO: Why handle temporary files differently? -** -** If the journal file is opened (or if it is already open), then a -** journal-header is written to the start of it. -** ** If the subjInMemory argument is non-zero, then any sub-journal opened ** within this transaction will be opened as an in-memory file. This ** has no effect if the sub-journal is already opened (as it may be when ** running in exclusive mode) or if the transaction does not require a ** sub-journal. If the subjInMemory argument is zero, then any required @@ -38617,24 +39679,24 @@ ** sub-journal is implemented in-memory if pPager is an in-memory database, ** or using a temporary file otherwise. */ SQLITE_PRIVATE int sqlite3PagerBegin(Pager *pPager, int exFlag, int subjInMemory){ int rc = SQLITE_OK; - assert( pPager->state!=PAGER_UNLOCK ); + + if( pPager->errCode ) return pPager->errCode; + assert( pPager->eState>=PAGER_READER && pPager->eState<PAGER_ERROR ); pPager->subjInMemory = (u8)subjInMemory; - if( pPager->state==PAGER_SHARED ){ + if( ALWAYS(pPager->eState==PAGER_READER) ){ assert( pPager->pInJournal==0 ); - assert( !MEMDB && !pPager->tempFile ); if( pagerUseWal(pPager) ){ /* If the pager is configured to use locking_mode=exclusive, and an ** exclusive lock on the database is not already held, obtain it now. */ if( pPager->exclusiveMode && sqlite3WalExclusiveMode(pPager->pWal, -1) ){ - rc = sqlite3OsLock(pPager->fd, EXCLUSIVE_LOCK); - pPager->state = PAGER_SHARED; + rc = pagerLockDb(pPager, EXCLUSIVE_LOCK); if( rc!=SQLITE_OK ){ return rc; } sqlite3WalExclusiveMode(pPager->pWal, 1); } @@ -38641,56 +39703,44 @@ /* Grab the write lock on the log file. If successful, upgrade to ** PAGER_RESERVED state. Otherwise, return an error code to the caller. ** The busy-handler is not invoked if another connection already ** holds the write-lock. If possible, the upper layer will call it. - ** - ** WAL mode sets Pager.state to PAGER_RESERVED when it has an open - ** transaction, but never to PAGER_EXCLUSIVE. This is because in - ** PAGER_EXCLUSIVE state the code to roll back savepoint transactions - ** may copy data from the sub-journal into the database file as well - ** as into the page cache. Which would be incorrect in WAL mode. */ rc = sqlite3WalBeginWriteTransaction(pPager->pWal); - if( rc==SQLITE_OK ){ - pPager->dbOrigSize = pPager->dbSize; - pPager->state = PAGER_RESERVED; - pPager->journalOff = 0; - } - - assert( rc!=SQLITE_OK || pPager->state==PAGER_RESERVED ); - assert( rc==SQLITE_OK || pPager->state==PAGER_SHARED ); }else{ /* Obtain a RESERVED lock on the database file. If the exFlag parameter ** is true, then immediately upgrade this to an EXCLUSIVE lock. The ** busy-handler callback can be used when upgrading to the EXCLUSIVE ** lock, but not when obtaining the RESERVED lock. */ - rc = sqlite3OsLock(pPager->fd, RESERVED_LOCK); - if( rc==SQLITE_OK ){ - pPager->state = PAGER_RESERVED; - if( exFlag ){ - rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); - } - } - } - - /* No need to open the journal file at this time. It will be - ** opened before it is written to. If we defer opening the journal, - ** we might save the work of creating a file if the transaction - ** ends up being a no-op. - */ - - if( rc!=SQLITE_OK ){ - assert( !pPager->dbModified ); - /* Ignore any IO error that occurs within pager_end_transaction(). The - ** purpose of this call is to reset the internal state of the pager - ** sub-system. It doesn't matter if the journal-file is not properly - ** finalized at this point (since it is not a valid journal file anyway). - */ - pager_end_transaction(pPager, 0); - } + rc = pagerLockDb(pPager, RESERVED_LOCK); + if( rc==SQLITE_OK && exFlag ){ + rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); + } + } + + if( rc==SQLITE_OK ){ + /* Change to WRITER_LOCKED state. + ** + ** WAL mode sets Pager.eState to PAGER_WRITER_LOCKED or CACHEMOD + ** when it has an open transaction, but never to DBMOD or FINISHED. + ** This is because in those states the code to roll back savepoint + ** transactions may copy data from the sub-journal into the database + ** file as well as into the page cache. Which would be incorrect in + ** WAL mode. + */ + pPager->eState = PAGER_WRITER_LOCKED; + pPager->dbHintSize = pPager->dbSize; + pPager->dbFileSize = pPager->dbSize; + pPager->dbOrigSize = pPager->dbSize; + pPager->journalOff = 0; + } + + assert( rc==SQLITE_OK || pPager->eState==PAGER_READER ); + assert( rc!=SQLITE_OK || pPager->eState==PAGER_WRITER_LOCKED ); + assert( assert_pager_state(pPager) ); } PAGERTRACE(("TRANSACTION %d\n", PAGERID(pPager))); return rc; } @@ -38705,110 +39755,98 @@ static int pager_write(PgHdr *pPg){ void *pData = pPg->pData; Pager *pPager = pPg->pPager; int rc = SQLITE_OK; - /* This routine is not called unless a transaction has already been - ** started. + /* This routine is not called unless a write-transaction has already + ** been started. The journal file may or may not be open at this point. + ** It is never called in the ERROR state. */ - assert( pPager->state>=PAGER_RESERVED ); + assert( pPager->eState==PAGER_WRITER_LOCKED + || pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + ); + assert( assert_pager_state(pPager) ); /* If an error has been previously detected, report the same error - ** again. - */ + ** again. This should not happen, but the check provides robustness. */ if( NEVER(pPager->errCode) ) return pPager->errCode; /* Higher-level routines never call this function if database is not ** writable. But check anyway, just for robustness. */ if( NEVER(pPager->readOnly) ) return SQLITE_PERM; - assert( !pPager->setMaster ); - CHECK_PAGE(pPg); + + /* The journal file needs to be opened. Higher level routines have already + ** obtained the necessary locks to begin the write-transaction, but the + ** rollback journal might not yet be open. Open it now if this is the case. + ** + ** This is done before calling sqlite3PcacheMakeDirty() on the page. + ** Otherwise, if it were done after calling sqlite3PcacheMakeDirty(), then + ** an error might occur and the pager would end up in WRITER_LOCKED state + ** with pages marked as dirty in the cache. + */ + if( pPager->eState==PAGER_WRITER_LOCKED ){ + rc = pager_open_journal(pPager); + if( rc!=SQLITE_OK ) return rc; + } + assert( pPager->eState>=PAGER_WRITER_CACHEMOD ); + assert( assert_pager_state(pPager) ); /* Mark the page as dirty. If the page has already been written ** to the journal then we can return right away. */ sqlite3PcacheMakeDirty(pPg); if( pageInJournal(pPg) && !subjRequiresPage(pPg) ){ assert( !pagerUseWal(pPager) ); - pPager->dbModified = 1; }else{ - - /* If we get this far, it means that the page needs to be - ** written to the transaction journal or the ckeckpoint journal - ** or both. - ** - ** Higher level routines should have already started a transaction, - ** which means they have acquired the necessary locks but the rollback - ** journal might not yet be open. - */ - assert( pPager->state>=RESERVED_LOCK ); - if( pPager->pInJournal==0 - && pPager->journalMode!=PAGER_JOURNALMODE_OFF - && !pagerUseWal(pPager) - ){ - assert( pPager->useJournal ); - rc = pager_open_journal(pPager); - if( rc!=SQLITE_OK ) return rc; - } - pPager->dbModified = 1; /* The transaction journal now exists and we have a RESERVED or an ** EXCLUSIVE lock on the main database file. Write the current page to ** the transaction journal if it is not there already. */ - if( !pageInJournal(pPg) && isOpen(pPager->jfd) ){ - assert( !pagerUseWal(pPager) ); - if( pPg->pgno<=pPager->dbOrigSize ){ + if( !pageInJournal(pPg) && !pagerUseWal(pPager) ){ + assert( pagerUseWal(pPager)==0 ); + if( pPg->pgno<=pPager->dbOrigSize && isOpen(pPager->jfd) ){ u32 cksum; char *pData2; + i64 iOff = pPager->journalOff; /* We should never write to the journal file the page that ** contains the database locks. The following assert verifies ** that we do not. */ assert( pPg->pgno!=PAGER_MJ_PGNO(pPager) ); - assert( pPager->journalHdr <= pPager->journalOff ); + assert( pPager->journalHdr<=pPager->journalOff ); CODEC2(pPager, pData, pPg->pgno, 7, return SQLITE_NOMEM, pData2); cksum = pager_cksum(pPager, (u8*)pData2); - rc = write32bits(pPager->jfd, pPager->journalOff, pPg->pgno); - if( rc==SQLITE_OK ){ - rc = sqlite3OsWrite(pPager->jfd, pData2, pPager->pageSize, - pPager->journalOff + 4); - pPager->journalOff += pPager->pageSize+4; - } - if( rc==SQLITE_OK ){ - rc = write32bits(pPager->jfd, pPager->journalOff, cksum); - pPager->journalOff += 4; - } + + /* Even if an IO or diskfull error occurs while journalling the + ** page in the block above, set the need-sync flag for the page. + ** Otherwise, when the transaction is rolled back, the logic in + ** playback_one_page() will think that the page needs to be restored + ** in the database file. And if an IO error occurs while doing so, + ** then corruption may follow. + */ + pPg->flags |= PGHDR_NEED_SYNC; + + rc = write32bits(pPager->jfd, iOff, pPg->pgno); + if( rc!=SQLITE_OK ) return rc; + rc = sqlite3OsWrite(pPager->jfd, pData2, pPager->pageSize, iOff+4); + if( rc!=SQLITE_OK ) return rc; + rc = write32bits(pPager->jfd, iOff+pPager->pageSize+4, cksum); + if( rc!=SQLITE_OK ) return rc; + IOTRACE(("JOUT %p %d %lld %d\n", pPager, pPg->pgno, pPager->journalOff, pPager->pageSize)); PAGER_INCR(sqlite3_pager_writej_count); PAGERTRACE(("JOURNAL %d page %d needSync=%d hash(%08x)\n", PAGERID(pPager), pPg->pgno, ((pPg->flags&PGHDR_NEED_SYNC)?1:0), pager_pagehash(pPg))); - /* Even if an IO or diskfull error occurred while journalling the - ** page in the block above, set the need-sync flag for the page. - ** Otherwise, when the transaction is rolled back, the logic in - ** playback_one_page() will think that the page needs to be restored - ** in the database file. And if an IO error occurs while doing so, - ** then corruption may follow. - */ - if( !pPager->noSync ){ - pPg->flags |= PGHDR_NEED_SYNC; - pPager->needSync = 1; - } - - /* An error has occurred writing to the journal file. The - ** transaction will be rolled back by the layer above. - */ - if( rc!=SQLITE_OK ){ - return rc; - } - + pPager->journalOff += 8 + pPager->pageSize; pPager->nRec++; assert( pPager->pInJournal!=0 ); rc = sqlite3BitvecSet(pPager->pInJournal, pPg->pgno); testcase( rc==SQLITE_NOMEM ); assert( rc==SQLITE_OK || rc==SQLITE_NOMEM ); @@ -38816,13 +39854,12 @@ if( rc!=SQLITE_OK ){ assert( rc==SQLITE_NOMEM ); return rc; } }else{ - if( !pPager->journalStarted && !pPager->noSync ){ + if( pPager->eState!=PAGER_WRITER_DBMOD ){ pPg->flags |= PGHDR_NEED_SYNC; - pPager->needSync = 1; } PAGERTRACE(("APPEND %d page %d needSync=%d\n", PAGERID(pPager), pPg->pgno, ((pPg->flags&PGHDR_NEED_SYNC)?1:0))); } @@ -38838,11 +39875,10 @@ } } /* Update the database size and return. */ - assert( pPager->state>=PAGER_SHARED ); if( pPager->dbSize<pPg->pgno ){ pPager->dbSize = pPg->pgno; } return rc; } @@ -38866,10 +39902,14 @@ PgHdr *pPg = pDbPage; Pager *pPager = pPg->pPager; Pgno nPagePerSector = (pPager->sectorSize/pPager->pageSize); + assert( pPager->eState>=PAGER_WRITER_LOCKED ); + assert( pPager->eState!=PAGER_ERROR ); + assert( assert_pager_state(pPager) ); + if( nPagePerSector>1 ){ Pgno nPageCount; /* Total number of pages in database file */ Pgno pg1; /* First page of the sector pPg is located on. */ int nPage = 0; /* Number of pages starting at pg1 to journal */ int ii; /* Loop counter */ @@ -38887,23 +39927,21 @@ ** an integer power of 2. It sets variable pg1 to the identifier ** of the first page of the sector pPg is located on. */ pg1 = ((pPg->pgno-1) & ~(nPagePerSector-1)) + 1; - rc = sqlite3PagerPagecount(pPager, (int *)&nPageCount); - if( rc==SQLITE_OK ){ - if( pPg->pgno>nPageCount ){ - nPage = (pPg->pgno - pg1)+1; - }else if( (pg1+nPagePerSector-1)>nPageCount ){ - nPage = nPageCount+1-pg1; - }else{ - nPage = nPagePerSector; - } - assert(nPage>0); - assert(pg1<=pPg->pgno); - assert((pg1+nPage)>pPg->pgno); - } + nPageCount = pPager->dbSize; + if( pPg->pgno>nPageCount ){ + nPage = (pPg->pgno - pg1)+1; + }else if( (pg1+nPagePerSector-1)>nPageCount ){ + nPage = nPageCount+1-pg1; + }else{ + nPage = nPagePerSector; + } + assert(nPage>0); + assert(pg1<=pPg->pgno); + assert((pg1+nPage)>pPg->pgno); for(ii=0; ii<nPage && rc==SQLITE_OK; ii++){ Pgno pg = pg1+ii; PgHdr *pPage; if( pg==pPg->pgno || !sqlite3BitvecTest(pPager->pInJournal, pg) ){ @@ -38911,11 +39949,10 @@ rc = sqlite3PagerGet(pPager, pg, &pPage); if( rc==SQLITE_OK ){ rc = pager_write(pPage); if( pPage->flags&PGHDR_NEED_SYNC ){ needSync = 1; - assert(pPager->needSync); } sqlite3PagerUnref(pPage); } } }else if( (pPage = pager_lookup(pPager, pg))!=0 ){ @@ -38931,19 +39968,18 @@ ** writing to any of these nPage pages may damage the others, the ** journal file must contain sync()ed copies of all of them ** before any of them can be written out to the database file. */ if( rc==SQLITE_OK && needSync ){ - assert( !MEMDB && pPager->noSync==0 ); + assert( !MEMDB ); for(ii=0; ii<nPage; ii++){ PgHdr *pPage = pager_lookup(pPager, pg1+ii); if( pPage ){ pPage->flags |= PGHDR_NEED_SYNC; sqlite3PagerUnref(pPage); } } - assert(pPager->needSync); } assert( pPager->doNotSyncSpill==1 ); pPager->doNotSyncSpill--; }else{ @@ -38981,13 +40017,11 @@ Pager *pPager = pPg->pPager; if( (pPg->flags&PGHDR_DIRTY) && pPager->nSavepoint==0 ){ PAGERTRACE(("DONT_WRITE page %d of %d\n", pPg->pgno, PAGERID(pPager))); IOTRACE(("CLEAN %p %d\n", pPager, pPg->pgno)) pPg->flags |= PGHDR_DONT_WRITE; -#ifdef SQLITE_CHECK_PAGES - pPg->pageHash = pager_pagehash(pPg); -#endif + pager_set_pagehash(pPg); } } /* ** This routine is called to increment the value of the database file @@ -39005,10 +40039,15 @@ ** by writing an updated version of page 1 using a call to the ** sqlite3OsWrite() function. */ static int pager_incr_changecounter(Pager *pPager, int isDirectMode){ int rc = SQLITE_OK; + + assert( pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + ); + assert( assert_pager_state(pPager) ); /* Declare and initialize constant integer 'isDirect'. If the ** atomic-write optimization is enabled in this build, then isDirect ** is initialized to the value passed as the isDirectMode parameter ** to this function. Otherwise, it is always set to zero. @@ -39024,11 +40063,10 @@ UNUSED_PARAMETER(isDirectMode); #else # define DIRECT_MODE isDirectMode #endif - assert( pPager->state>=PAGER_RESERVED ); if( !pPager->changeCountDone && pPager->dbSize>0 ){ PgHdr *pPgHdr; /* Reference to page 1 */ u32 change_counter; /* Initial value of change-counter field */ assert( !pPager->tempFile && isOpen(pPager->fd) ); @@ -39109,13 +40147,17 @@ ** Otherwise, either SQLITE_BUSY or an SQLITE_IOERR_XXX error code is ** returned. */ SQLITE_PRIVATE int sqlite3PagerExclusiveLock(Pager *pPager){ int rc = SQLITE_OK; - assert( pPager->state>=PAGER_RESERVED ); + assert( pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + || pPager->eState==PAGER_WRITER_LOCKED + ); + assert( assert_pager_state(pPager) ); if( 0==pagerUseWal(pPager) ){ - rc = pager_wait_on_lock(pPager, PAGER_EXCLUSIVE); + rc = pager_wait_on_lock(pPager, EXCLUSIVE_LOCK); } return rc; } /* @@ -39149,26 +40191,33 @@ const char *zMaster, /* If not NULL, the master journal name */ int noSync /* True to omit the xSync on the db file */ ){ int rc = SQLITE_OK; /* Return code */ - /* The dbOrigSize is never set if journal_mode=OFF */ - assert( pPager->journalMode!=PAGER_JOURNALMODE_OFF || pPager->dbOrigSize==0 ); + assert( pPager->eState==PAGER_WRITER_LOCKED + || pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + || pPager->eState==PAGER_ERROR + ); + assert( assert_pager_state(pPager) ); /* If a prior error occurred, report that error again. */ - if( pPager->errCode ) return pPager->errCode; + if( NEVER(pPager->errCode) ) return pPager->errCode; PAGERTRACE(("DATABASE SYNC: File=%s zMaster=%s nSize=%d\n", pPager->zFilename, zMaster, pPager->dbSize)); - if( MEMDB && pPager->dbModified ){ + /* If no database changes have been made, return early. */ + if( pPager->eState<PAGER_WRITER_CACHEMOD ) return SQLITE_OK; + + if( MEMDB ){ /* If this is an in-memory db, or no pages have been written to, or this ** function has already been called, it is mostly a no-op. However, any ** backup in progress needs to be restarted. */ sqlite3BackupRestart(pPager->pBackup); - }else if( pPager->dbModified ){ + }else{ if( pagerUseWal(pPager) ){ PgHdr *pList = sqlite3PcacheDirtyList(pPager->pPCache); if( pList ){ rc = pagerWalFrames(pPager, pList, pPager->dbSize, 1, (pPager->fullSync ? pPager->sync_flags : 0) @@ -39207,11 +40256,11 @@ || pPager->journalMode==PAGER_JOURNALMODE_OFF || pPager->journalMode==PAGER_JOURNALMODE_WAL ); if( !zMaster && isOpen(pPager->jfd) && pPager->journalOff==jrnlBufferSize(pPager) - && pPager->dbSize>=pPager->dbFileSize + && pPager->dbSize>=pPager->dbOrigSize && (0==(pPg = sqlite3PcacheDirtyList(pPager->pPCache)) || 0==pPg->pDirty) ){ /* Update the db file change counter via the direct-write method. The ** following call will modify the in-memory representation of page 1 ** to include the updated change counter and then write page 1 @@ -39237,17 +40286,14 @@ ** Before reading the pages with page numbers larger than the ** current value of Pager.dbSize, set dbSize back to the value ** that it took at the start of the transaction. Otherwise, the ** calls to sqlite3PagerGet() return zeroed pages instead of ** reading data from the database file. - ** - ** When journal_mode==OFF the dbOrigSize is always zero, so this - ** block never runs if journal_mode=OFF. */ #ifndef SQLITE_OMIT_AUTOVACUUM if( pPager->dbSize<pPager->dbOrigSize - && ALWAYS(pPager->journalMode!=PAGER_JOURNALMODE_OFF) + && pPager->journalMode!=PAGER_JOURNALMODE_OFF ){ Pgno i; /* Iterator variable */ const Pgno iSkip = PAGER_MJ_PGNO(pPager); /* Pending lock page */ const Pgno dbSize = pPager->dbSize; /* Database image size */ pPager->dbSize = pPager->dbOrigSize; @@ -39270,18 +40316,24 @@ ** or if zMaster is NULL (no master journal), then this call is a no-op. */ rc = writeMasterJournal(pPager, zMaster); if( rc!=SQLITE_OK ) goto commit_phase_one_exit; - /* Sync the journal file. If the atomic-update optimization is being - ** used, this call will not create the journal file or perform any - ** real IO. + /* Sync the journal file and write all dirty pages to the database. + ** If the atomic-update optimization is being used, this sync will not + ** create the journal file or perform any real IO. + ** + ** Because the change-counter page was just modified, unless the + ** atomic-update optimization is used it is almost certain that the + ** journal requires a sync here. However, in locking_mode=exclusive + ** on a system under memory pressure it is just possible that this is + ** not the case. In this case it is likely enough that the redundant + ** xSync() call will be changed to a no-op by the OS anyhow. */ - rc = syncJournal(pPager); + rc = syncJournal(pPager, 0); if( rc!=SQLITE_OK ) goto commit_phase_one_exit; - /* Write all dirty pages to the database file. */ rc = pager_write_pagelist(pPager,sqlite3PcacheDirtyList(pPager->pPCache)); if( rc!=SQLITE_OK ){ assert( rc!=SQLITE_IOERR_BLOCKED ); goto commit_phase_one_exit; } @@ -39290,11 +40342,11 @@ /* If the file on disk is not the same size as the database image, ** then use pager_truncate to grow or shrink the file here. */ if( pPager->dbSize!=pPager->dbFileSize ){ Pgno nNew = pPager->dbSize - (pPager->dbSize==PAGER_MJ_PGNO(pPager)); - assert( pPager->state>=PAGER_EXCLUSIVE ); + assert( pPager->eState==PAGER_WRITER_DBMOD ); rc = pager_truncate(pPager, nNew); if( rc!=SQLITE_OK ) goto commit_phase_one_exit; } /* Finally, sync the database file. */ @@ -39301,16 +40353,16 @@ if( !pPager->noSync && !noSync ){ rc = sqlite3OsSync(pPager->fd, pPager->sync_flags); } IOTRACE(("DBSYNC %p\n", pPager)) } - - assert( pPager->state!=PAGER_SYNCED ); - pPager->state = PAGER_SYNCED; } commit_phase_one_exit: + if( rc==SQLITE_OK && !pagerUseWal(pPager) ){ + pPager->eState = PAGER_WRITER_FINISHED; + } return rc; } /* @@ -39334,16 +40386,15 @@ /* This routine should not be called if a prior error has occurred. ** But if (due to a coding error elsewhere in the system) it does get ** called, just return the same error code without doing anything. */ if( NEVER(pPager->errCode) ) return pPager->errCode; - /* This function should not be called if the pager is not in at least - ** PAGER_RESERVED state. **FIXME**: Make it so that this test always - ** fails - make it so that we never reach this point if we do not hold - ** all necessary locks. - */ - if( NEVER(pPager->state<PAGER_RESERVED) ) return SQLITE_ERROR; + assert( pPager->eState==PAGER_WRITER_LOCKED + || pPager->eState==PAGER_WRITER_FINISHED + || (pagerUseWal(pPager) && pPager->eState==PAGER_WRITER_CACHEMOD) + ); + assert( assert_pager_state(pPager) ); /* An optimization. If the database was not actually modified during ** this transaction, the pager is running in exclusive-mode and is ** using persistent journals, then this function is a no-op. ** @@ -39352,106 +40403,80 @@ ** a hot-journal during hot-journal rollback, 0 changes will be made ** to the database file. So there is no need to zero the journal ** header. Since the pager is in exclusive mode, there is no need ** to drop any locks either. */ - if( pPager->dbModified==0 && pPager->exclusiveMode + if( pPager->eState==PAGER_WRITER_LOCKED + && pPager->exclusiveMode && pPager->journalMode==PAGER_JOURNALMODE_PERSIST ){ assert( pPager->journalOff==JOURNAL_HDR_SZ(pPager) || !pPager->journalOff ); + pPager->eState = PAGER_READER; return SQLITE_OK; } PAGERTRACE(("COMMIT %d\n", PAGERID(pPager))); - assert( pPager->state==PAGER_SYNCED || MEMDB || !pPager->dbModified ); rc = pager_end_transaction(pPager, pPager->setMaster); return pager_error(pPager, rc); } /* -** Rollback all changes. The database falls back to PAGER_SHARED mode. +** If a write transaction is open, then all changes made within the +** transaction are reverted and the current write-transaction is closed. +** The pager falls back to PAGER_READER state if successful, or PAGER_ERROR +** state if an error occurs. ** -** This function performs two tasks: +** If the pager is already in PAGER_ERROR state when this function is called, +** it returns Pager.errCode immediately. No work is performed in this case. +** +** Otherwise, in rollback mode, this function performs two functions: ** ** 1) It rolls back the journal file, restoring all database file and ** in-memory cache pages to the state they were in when the transaction ** was opened, and +** ** 2) It finalizes the journal file, so that it is not used for hot ** rollback at any point in the future. ** -** subject to the following qualifications: -** -** * If the journal file is not yet open when this function is called, -** then only (2) is performed. In this case there is no journal file -** to roll back. -** -** * If in an error state other than SQLITE_FULL, then task (1) is -** performed. If successful, task (2). Regardless of the outcome -** of either, the error state error code is returned to the caller -** (i.e. either SQLITE_IOERR or SQLITE_CORRUPT). -** -** * If the pager is in PAGER_RESERVED state, then attempt (1). Whether -** or not (1) is successful, also attempt (2). If successful, return -** SQLITE_OK. Otherwise, enter the error state and return the first -** error code encountered. -** -** In this case there is no chance that the database was written to. -** So is safe to finalize the journal file even if the playback -** (operation 1) failed. However the pager must enter the error state -** as the contents of the in-memory cache are now suspect. -** -** * Finally, if in PAGER_EXCLUSIVE state, then attempt (1). Only -** attempt (2) if (1) is successful. Return SQLITE_OK if successful, -** otherwise enter the error state and return the error code from the -** failing operation. -** -** In this case the database file may have been written to. So if the -** playback operation did not succeed it would not be safe to finalize -** the journal file. It needs to be left in the file-system so that -** some other process can use it to restore the database state (by -** hot-journal rollback). +** Finalization of the journal file (task 2) is only performed if the +** rollback is successful. +** +** In WAL mode, all cache-entries containing data modified within the +** current transaction are either expelled from the cache or reverted to +** their pre-transaction state by re-reading data from the database or +** WAL files. The WAL transaction is then closed. */ SQLITE_PRIVATE int sqlite3PagerRollback(Pager *pPager){ int rc = SQLITE_OK; /* Return code */ PAGERTRACE(("ROLLBACK %d\n", PAGERID(pPager))); + + /* PagerRollback() is a no-op if called in READER or OPEN state. If + ** the pager is already in the ERROR state, the rollback is not + ** attempted here. Instead, the error code is returned to the caller. + */ + assert( assert_pager_state(pPager) ); + if( pPager->eState==PAGER_ERROR ) return pPager->errCode; + if( pPager->eState<=PAGER_READER ) return SQLITE_OK; + if( pagerUseWal(pPager) ){ int rc2; - rc = sqlite3PagerSavepoint(pPager, SAVEPOINT_ROLLBACK, -1); rc2 = pager_end_transaction(pPager, pPager->setMaster); if( rc==SQLITE_OK ) rc = rc2; - rc = pager_error(pPager, rc); - }else if( !pPager->dbModified || !isOpen(pPager->jfd) ){ - rc = pager_end_transaction(pPager, pPager->setMaster); - }else if( pPager->errCode && pPager->errCode!=SQLITE_FULL ){ - if( pPager->state>=PAGER_EXCLUSIVE ){ - pager_playback(pPager, 0); - } - rc = pPager->errCode; - }else{ - if( pPager->state==PAGER_RESERVED ){ - int rc2; - rc = pager_playback(pPager, 0); - rc2 = pager_end_transaction(pPager, pPager->setMaster); - if( rc==SQLITE_OK ){ - rc = rc2; - } - }else{ - rc = pager_playback(pPager, 0); - } - - if( !MEMDB ){ - pPager->dbSizeValid = 0; - } - - /* If an error occurs during a ROLLBACK, we can no longer trust the pager - ** cache. So call pager_error() on the way out to make any error - ** persistent. - */ - rc = pager_error(pPager, rc); - } - return rc; + }else if( !isOpen(pPager->jfd) || pPager->eState==PAGER_WRITER_LOCKED ){ + rc = pager_end_transaction(pPager, 0); + }else{ + rc = pager_playback(pPager, 0); + } + + assert( pPager->eState==PAGER_READER || rc!=SQLITE_OK ); + assert( rc==SQLITE_OK || rc==SQLITE_FULL || (rc&0xFF)==SQLITE_IOERR ); + + /* If an error occurs during a ROLLBACK, we can no longer trust the pager + ** cache. So call pager_error() on the way out to make any error persistent. + */ + return pager_error(pPager, rc); } /* ** Return TRUE if the database file is opened read-only. Return FALSE ** if the database is (in theory) writable. @@ -39493,12 +40518,12 @@ SQLITE_PRIVATE int *sqlite3PagerStats(Pager *pPager){ static int a[11]; a[0] = sqlite3PcacheRefCount(pPager->pPCache); a[1] = sqlite3PcachePagecount(pPager->pPCache); a[2] = sqlite3PcacheGetCachesize(pPager->pPCache); - a[3] = pPager->dbSizeValid ? (int) pPager->dbSize : -1; - a[4] = pPager->state; + a[3] = pPager->eState==PAGER_OPEN ? -1 : (int) pPager->dbSize; + a[4] = pPager->eState; a[5] = pPager->errCode; a[6] = pPager->nHit; a[7] = pPager->nMiss; a[8] = 0; /* Used to be pPager->nOvfl */ a[9] = pPager->nRead; @@ -39525,18 +40550,17 @@ ** returned. Otherwise, SQLITE_OK. */ SQLITE_PRIVATE int sqlite3PagerOpenSavepoint(Pager *pPager, int nSavepoint){ int rc = SQLITE_OK; /* Return code */ int nCurrent = pPager->nSavepoint; /* Current number of savepoints */ + + assert( pPager->eState>=PAGER_WRITER_LOCKED ); + assert( assert_pager_state(pPager) ); if( nSavepoint>nCurrent && pPager->useJournal ){ int ii; /* Iterator variable */ PagerSavepoint *aNew; /* New Pager.aSavepoint array */ - int nPage; /* Size of database file */ - - rc = sqlite3PagerPagecount(pPager, &nPage); - if( rc ) return rc; /* Grow the Pager.aSavepoint array using realloc(). Return SQLITE_NOMEM ** if the allocation fails. Otherwise, zero the new portion in case a ** malloc failure occurs while populating it in the for(...) loop below. */ @@ -39549,18 +40573,18 @@ memset(&aNew[nCurrent], 0, (nSavepoint-nCurrent) * sizeof(PagerSavepoint)); pPager->aSavepoint = aNew; /* Populate the PagerSavepoint structures just allocated. */ for(ii=nCurrent; ii<nSavepoint; ii++){ - aNew[ii].nOrig = nPage; + aNew[ii].nOrig = pPager->dbSize; if( isOpen(pPager->jfd) && pPager->journalOff>0 ){ aNew[ii].iOffset = pPager->journalOff; }else{ aNew[ii].iOffset = JOURNAL_HDR_SZ(pPager); } aNew[ii].iSubRec = pPager->nSubRec; - aNew[ii].pInSavepoint = sqlite3BitvecCreate(nPage); + aNew[ii].pInSavepoint = sqlite3BitvecCreate(pPager->dbSize); if( !aNew[ii].pInSavepoint ){ return SQLITE_NOMEM; } if( pagerUseWal(pPager) ){ sqlite3WalSavepoint(pPager->pWal, aNew[ii].aWalData); @@ -39603,16 +40627,16 @@ ** This function may return SQLITE_NOMEM if a memory allocation fails, ** or an IO error code if an IO error occurs while rolling back a ** savepoint. If no errors occur, SQLITE_OK is returned. */ SQLITE_PRIVATE int sqlite3PagerSavepoint(Pager *pPager, int op, int iSavepoint){ - int rc = SQLITE_OK; + int rc = pPager->errCode; /* Return code */ assert( op==SAVEPOINT_RELEASE || op==SAVEPOINT_ROLLBACK ); assert( iSavepoint>=0 || op==SAVEPOINT_ROLLBACK ); - if( iSavepoint<pPager->nSavepoint ){ + if( rc==SQLITE_OK && iSavepoint<pPager->nSavepoint ){ int ii; /* Iterator variable */ int nNew; /* Number of remaining savepoints after this op. */ /* Figure out how many savepoints will still be active after this ** operation. Store this value in nNew. Then free resources associated @@ -39644,12 +40668,12 @@ else if( pagerUseWal(pPager) || isOpen(pPager->jfd) ){ PagerSavepoint *pSavepoint = (nNew==0)?0:&pPager->aSavepoint[nNew-1]; rc = pagerPlaybackSavepoint(pPager, pSavepoint); assert(rc!=SQLITE_DONE); } - } + return rc; } /* ** Return the full pathname of the database file. @@ -39743,10 +40767,14 @@ Pgno needSyncPgno = 0; /* Old value of pPg->pgno, if sync is required */ int rc; /* Return code */ Pgno origPgno; /* The original page number */ assert( pPg->nRef>0 ); + assert( pPager->eState==PAGER_WRITER_CACHEMOD + || pPager->eState==PAGER_WRITER_DBMOD + ); + assert( assert_pager_state(pPager) ); /* In order to be able to rollback, an in-memory database must journal ** the page we are moving from. */ if( MEMDB ){ @@ -39792,15 +40820,14 @@ */ if( (pPg->flags&PGHDR_NEED_SYNC) && !isCommit ){ needSyncPgno = pPg->pgno; assert( pageInJournal(pPg) || pPg->pgno>pPager->dbOrigSize ); assert( pPg->flags&PGHDR_DIRTY ); - assert( pPager->needSync ); } /* If the cache contains a page with page-number pgno, remove it - ** from its hash chain. Also, if the PgHdr.needSync was set for + ** from its hash chain. Also, if the PGHDR_NEED_SYNC flag was set for ** page pgno before the 'move' operation, it needs to be retained ** for the page moved there. */ pPg->flags &= ~PGHDR_NEED_SYNC; pPgOld = pager_lookup(pPager, pgno); @@ -39808,67 +40835,59 @@ if( pPgOld ){ pPg->flags |= (pPgOld->flags&PGHDR_NEED_SYNC); if( MEMDB ){ /* Do not discard pages from an in-memory database since we might ** need to rollback later. Just move the page out of the way. */ - assert( pPager->dbSizeValid ); sqlite3PcacheMove(pPgOld, pPager->dbSize+1); }else{ sqlite3PcacheDrop(pPgOld); } } origPgno = pPg->pgno; sqlite3PcacheMove(pPg, pgno); sqlite3PcacheMakeDirty(pPg); - pPager->dbModified = 1; + + /* For an in-memory database, make sure the original page continues + ** to exist, in case the transaction needs to roll back. Use pPgOld + ** as the original page since it has already been allocated. + */ + if( MEMDB ){ + assert( pPgOld ); + sqlite3PcacheMove(pPgOld, origPgno); + sqlite3PagerUnref(pPgOld); + } if( needSyncPgno ){ /* If needSyncPgno is non-zero, then the journal file needs to be ** sync()ed before any data is written to database file page needSyncPgno. ** Currently, no such page exists in the page-cache and the ** "is journaled" bitvec flag has been set. This needs to be remedied by - ** loading the page into the pager-cache and setting the PgHdr.needSync + ** loading the page into the pager-cache and setting the PGHDR_NEED_SYNC ** flag. ** ** If the attempt to load the page into the page-cache fails, (due ** to a malloc() or IO failure), clear the bit in the pInJournal[] ** array. Otherwise, if the page is loaded and written again in ** this transaction, it may be written to the database file before ** it is synced into the journal file. This way, it may end up in ** the journal file twice, but that is not a problem. - ** - ** The sqlite3PagerGet() call may cause the journal to sync. So make - ** sure the Pager.needSync flag is set too. */ PgHdr *pPgHdr; - assert( pPager->needSync ); rc = sqlite3PagerGet(pPager, needSyncPgno, &pPgHdr); if( rc!=SQLITE_OK ){ if( needSyncPgno<=pPager->dbOrigSize ){ assert( pPager->pTmpSpace!=0 ); sqlite3BitvecClear(pPager->pInJournal, needSyncPgno, pPager->pTmpSpace); } return rc; } - pPager->needSync = 1; - assert( pPager->noSync==0 && !MEMDB ); pPgHdr->flags |= PGHDR_NEED_SYNC; sqlite3PcacheMakeDirty(pPgHdr); sqlite3PagerUnref(pPgHdr); } - /* - ** For an in-memory database, make sure the original page continues - ** to exist, in case the transaction needs to roll back. Use pPgOld - ** as the original page since it has already been allocated. - */ - if( MEMDB ){ - sqlite3PcacheMove(pPgOld, origPgno); - sqlite3PagerUnref(pPgOld); - } - return SQLITE_OK; } #endif /* @@ -39929,10 +40948,17 @@ ** ** The returned indicate the current (possibly updated) journal-mode. */ SQLITE_PRIVATE int sqlite3PagerSetJournalMode(Pager *pPager, int eMode){ u8 eOld = pPager->journalMode; /* Prior journalmode */ + +#ifdef SQLITE_DEBUG + /* The print_pager_state() routine is intended to be used by the debugger + ** only. We invoke it once here to suppress a compiler warning. */ + print_pager_state(pPager); +#endif + /* The eMode parameter is always valid */ assert( eMode==PAGER_JOURNALMODE_DELETE || eMode==PAGER_JOURNALMODE_TRUNCATE || eMode==PAGER_JOURNALMODE_PERSIST @@ -39955,24 +40981,17 @@ eMode = eOld; } } if( eMode!=eOld ){ - /* When changing between rollback modes, close the journal file prior - ** to the change. But when changing from a rollback mode to WAL, keep - ** the journal open since there is a rollback-style transaction in play - ** used to convert the version numbers in the btree header. - */ - if( isOpen(pPager->jfd) && eMode!=PAGER_JOURNALMODE_WAL ){ - sqlite3OsClose(pPager->jfd); - } /* Change the journal mode. */ + assert( pPager->eState!=PAGER_ERROR ); pPager->journalMode = (u8)eMode; /* When transistioning from TRUNCATE or PERSIST to any other journal - ** mode except WAL (and we are not in locking_mode=EXCLUSIVE) then + ** mode except WAL, unless the pager is in locking_mode=exclusive mode, ** delete the journal file. */ assert( (PAGER_JOURNALMODE_TRUNCATE & 5)==1 ); assert( (PAGER_JOURNALMODE_PERSIST & 5)==1 ); assert( (PAGER_JOURNALMODE_DELETE & 5)==0 ); @@ -39989,28 +41008,34 @@ ** ** Before deleting the journal file, obtain a RESERVED lock on the ** database file. This ensures that the journal file is not deleted ** while it is in use by some other client. */ - int rc = SQLITE_OK; - int state = pPager->state; - if( state<PAGER_SHARED ){ - rc = sqlite3PagerSharedLock(pPager); - } - if( pPager->state==PAGER_SHARED ){ - assert( rc==SQLITE_OK ); - rc = sqlite3OsLock(pPager->fd, RESERVED_LOCK); - } - if( rc==SQLITE_OK ){ + sqlite3OsClose(pPager->jfd); + if( pPager->eLock>=RESERVED_LOCK ){ sqlite3OsDelete(pPager->pVfs, pPager->zJournal, 0); + }else{ + int rc = SQLITE_OK; + int state = pPager->eState; + assert( state==PAGER_OPEN || state==PAGER_READER ); + if( state==PAGER_OPEN ){ + rc = sqlite3PagerSharedLock(pPager); + } + if( pPager->eState==PAGER_READER ){ + assert( rc==SQLITE_OK ); + rc = pagerLockDb(pPager, RESERVED_LOCK); + } + if( rc==SQLITE_OK ){ + sqlite3OsDelete(pPager->pVfs, pPager->zJournal, 0); + } + if( rc==SQLITE_OK && state==PAGER_READER ){ + pagerUnlockDb(pPager, SHARED_LOCK); + }else if( state==PAGER_OPEN ){ + pager_unlock(pPager); + } + assert( state==pPager->eState ); } - if( rc==SQLITE_OK && state==PAGER_SHARED ){ - sqlite3OsUnlock(pPager->fd, SHARED_LOCK); - }else if( state==PAGER_UNLOCK ){ - pager_unlock(pPager); - } - assert( state==pPager->state ); } } /* Return the new journal mode */ return (int)pPager->journalMode; @@ -40027,11 +41052,12 @@ ** Return TRUE if the pager is in a state where it is OK to change the ** journalmode. Journalmode changes can only happen when the database ** is unmodified. */ SQLITE_PRIVATE int sqlite3PagerOkToChangeJournalMode(Pager *pPager){ - if( pPager->dbModified ) return 0; + assert( assert_pager_state(pPager) ); + if( pPager->eState>=PAGER_WRITER_CACHEMOD ) return 0; if( NEVER(isOpen(pPager->jfd) && pPager->journalOff>0) ) return 0; return 1; } /* @@ -40092,39 +41118,46 @@ ** ** If the pager passed as the first argument is open on a real database ** file (not a temp file or an in-memory database), and the WAL file ** is not already open, make an attempt to open it now. If successful, ** return SQLITE_OK. If an error occurs or the VFS used by the pager does -** not support the xShmXXX() methods, return an error code. *pisOpen is +** not support the xShmXXX() methods, return an error code. *pbOpen is ** not modified in either case. ** ** If the pager is open on a temp-file (or in-memory database), or if -** the WAL file is already open, set *pisOpen to 1 and return SQLITE_OK +** the WAL file is already open, set *pbOpen to 1 and return SQLITE_OK ** without doing anything. */ SQLITE_PRIVATE int sqlite3PagerOpenWal( Pager *pPager, /* Pager object */ - int *pisOpen /* OUT: Set to true if call is a no-op */ + int *pbOpen /* OUT: Set to true if call is a no-op */ ){ int rc = SQLITE_OK; /* Return code */ - assert( pPager->state>=PAGER_SHARED ); - assert( (pisOpen==0 && !pPager->tempFile && !pPager->pWal) || *pisOpen==0 ); + assert( assert_pager_state(pPager) ); + assert( pPager->eState==PAGER_OPEN || pbOpen ); + assert( pPager->eState==PAGER_READER || !pbOpen ); + assert( pbOpen==0 || *pbOpen==0 ); + assert( pbOpen!=0 || (!pPager->tempFile && !pPager->pWal) ); if( !pPager->tempFile && !pPager->pWal ){ if( !sqlite3PagerWalSupported(pPager) ) return SQLITE_CANTOPEN; + + /* Close any rollback journal previously open */ + sqlite3OsClose(pPager->jfd); /* Open the connection to the log file. If this operation fails, ** (e.g. due to malloc() failure), unlock the database file and ** return an error code. */ rc = sqlite3WalOpen(pPager->pVfs, pPager->fd, pPager->zWal, &pPager->pWal); if( rc==SQLITE_OK ){ pPager->journalMode = PAGER_JOURNALMODE_WAL; + pPager->eState = PAGER_OPEN; } }else{ - *pisOpen = 1; + *pbOpen = 1; } return rc; } @@ -40146,11 +41179,11 @@ ** it may need to be checkpointed before the connection can switch to ** rollback mode. Open it now so this can happen. */ if( !pPager->pWal ){ int logexists = 0; - rc = sqlite3OsLock(pPager->fd, SQLITE_LOCK_SHARED); + rc = pagerLockDb(pPager, SHARED_LOCK); if( rc==SQLITE_OK ){ rc = sqlite3OsAccess( pPager->pVfs, pPager->zWal, SQLITE_ACCESS_EXISTS, &logexists ); } @@ -40162,21 +41195,21 @@ /* Checkpoint and close the log. Because an EXCLUSIVE lock is held on ** the database file, the log and log-summary files will be deleted. */ if( rc==SQLITE_OK && pPager->pWal ){ - rc = sqlite3OsLock(pPager->fd, SQLITE_LOCK_EXCLUSIVE); + rc = pagerLockDb(pPager, EXCLUSIVE_LOCK); if( rc==SQLITE_OK ){ rc = sqlite3WalClose(pPager->pWal, (pPager->noSync ? 0 : pPager->sync_flags), pPager->pageSize, (u8*)pPager->pTmpSpace ); pPager->pWal = 0; }else{ /* If we cannot get an EXCLUSIVE lock, downgrade the PENDING lock ** that we did get back to SHARED. */ - sqlite3OsUnlock(pPager->fd, SQLITE_LOCK_SHARED); + pagerUnlockDb(pPager, SQLITE_LOCK_SHARED); } } return rc; } @@ -40492,18 +41525,22 @@ /* ** The following object holds a copy of the wal-index header content. ** ** The actual header in the wal-index consists of two copies of this ** object. +** +** The szPage value can be any power of 2 between 512 and 32768, inclusive. +** Or it can be 1 to represent a 65536-byte page. The latter case was +** added in 3.7.1 when support for 64K pages was added. */ struct WalIndexHdr { u32 iVersion; /* Wal-index version */ u32 unused; /* Unused (padding) field */ u32 iChange; /* Counter incremented each transaction */ u8 isInit; /* 1 when initialized */ u8 bigEndCksum; /* True if checksums in WAL are big-endian */ - u16 szPage; /* Database page size in bytes */ + u16 szPage; /* Database page size in bytes. 1==64K */ u32 mxFrame; /* Index of last valid frame in the WAL */ u32 nPage; /* Size of database in pages */ u32 aFrameCksum[2]; /* Checksum of last frame in log */ u32 aSalt[2]; /* Two salt values copied from WAL header */ u32 aCksum[2]; /* Checksum over all prior fields */ @@ -40610,11 +41647,11 @@ sqlite3_file *pDbFd; /* File handle for the database file */ sqlite3_file *pWalFd; /* File handle for WAL file */ u32 iCallback; /* Value to pass to log callback (or 0) */ int nWiData; /* Size of array apWiData */ volatile u32 **apWiData; /* Pointer to wal-index content in memory */ - u16 szPage; /* Database page size */ + u32 szPage; /* Database page size */ i16 readLock; /* Which read lock is being held. -1 for none */ u8 exclusiveMode; /* Non-zero if connection is in exclusive mode */ u8 writeLock; /* True if in a write transaction */ u8 ckptLock; /* True if holding a checkpoint lock */ u8 readOnly; /* True if the WAL file is open read-only */ @@ -41281,11 +42318,11 @@ || szPage<512 ){ goto finished; } pWal->hdr.bigEndCksum = (u8)(magic&0x00000001); - pWal->szPage = (u16)szPage; + pWal->szPage = szPage; pWal->nCkpt = sqlite3Get4byte(&aBuf[12]); memcpy(&pWal->hdr.aSalt, &aBuf[16], 8); /* Verify that the WAL header checksum is correct */ walChecksumBytes(pWal->hdr.bigEndCksum==SQLITE_BIGENDIAN, @@ -41331,11 +42368,13 @@ /* If nTruncate is non-zero, this is a commit record. */ if( nTruncate ){ pWal->hdr.mxFrame = iFrame; pWal->hdr.nPage = nTruncate; - pWal->hdr.szPage = (u16)szPage; + pWal->hdr.szPage = (u16)((szPage&0xff00) | (szPage>>16)); + testcase( szPage<=32768 ); + testcase( szPage>=65536 ); aFrameCksum[0] = pWal->hdr.aFrameCksum[0]; aFrameCksum[1] = pWal->hdr.aFrameCksum[1]; } } @@ -41356,10 +42395,21 @@ */ pInfo = walCkptInfo(pWal); pInfo->nBackfill = 0; pInfo->aReadMark[0] = 0; for(i=1; i<WAL_NREADER; i++) pInfo->aReadMark[i] = READMARK_NOT_USED; + + /* If more than one frame was recovered from the log file, report an + ** event via sqlite3_log(). This is to help with identifying performance + ** problems caused by applications routinely shutting down without + ** checkpointing the log file. + */ + if( pWal->hdr.nPage ){ + sqlite3_log(SQLITE_OK, "Recovered %d frames from WAL file %s", + pWal->hdr.nPage, pWal->zWalName + ); + } } recovery_error: WALTRACE(("WAL%p: recovery %s\n", pWal, rc ? "failed" : "ok")); walUnlockExclusive(pWal, iLock, nLock); @@ -41716,18 +42766,22 @@ int sync_flags, /* Flags for OsSync() (or 0) */ int nBuf, /* Size of zBuf in bytes */ u8 *zBuf /* Temporary buffer to use */ ){ int rc; /* Return code */ - int szPage = pWal->hdr.szPage; /* Database page-size */ + int szPage; /* Database page-size */ WalIterator *pIter = 0; /* Wal iterator context */ u32 iDbpage = 0; /* Next database page to write */ u32 iFrame = 0; /* Wal frame containing data for iDbpage */ u32 mxSafeFrame; /* Max frame that can be backfilled */ + u32 mxPage; /* Max database page to write */ int i; /* Loop counter */ volatile WalCkptInfo *pInfo; /* The checkpoint status information */ + szPage = (pWal->hdr.szPage&0xfe00) + ((pWal->hdr.szPage&0x0001)<<16); + testcase( szPage<=32768 ); + testcase( szPage>=65536 ); if( pWal->hdr.mxFrame==0 ) return SQLITE_OK; /* Allocate the iterator */ rc = walIteratorInit(pWal, &pIter); if( rc!=SQLITE_OK ){ @@ -41734,11 +42788,11 @@ return rc; } assert( pIter ); /*** TODO: Move this test out to the caller. Make it an assert() here ***/ - if( pWal->hdr.szPage!=nBuf ){ + if( szPage!=nBuf ){ rc = SQLITE_CORRUPT_BKPT; goto walcheckpoint_out; } /* Compute in mxSafeFrame the index of the last frame of the WAL that is @@ -41745,10 +42799,11 @@ ** safe to write into the database. Frames beyond mxSafeFrame might ** overwrite database pages that are in use by active readers and thus ** cannot be backfilled from the WAL. */ mxSafeFrame = pWal->hdr.mxFrame; + mxPage = pWal->hdr.nPage; pInfo = walCkptInfo(pWal); for(i=1; i<WAL_NREADER; i++){ u32 y = pInfo->aReadMark[i]; if( mxSafeFrame>=y ){ assert( y<=pWal->hdr.mxFrame ); @@ -41765,22 +42820,34 @@ } if( pInfo->nBackfill<mxSafeFrame && (rc = walLockExclusive(pWal, WAL_READ_LOCK(0), 1))==SQLITE_OK ){ + i64 nSize; /* Current size of database file */ u32 nBackfill = pInfo->nBackfill; /* Sync the WAL to disk */ if( sync_flags ){ rc = sqlite3OsSync(pWal->pWalFd, sync_flags); } + + /* If the database file may grow as a result of this checkpoint, hint + ** about the eventual size of the db file to the VFS layer. + */ + if( rc==SQLITE_OK ){ + i64 nReq = ((i64)mxPage * szPage); + rc = sqlite3OsFileSize(pWal->pDbFd, &nSize); + if( rc==SQLITE_OK && nSize<nReq ){ + sqlite3OsFileControl(pWal->pDbFd, SQLITE_FCNTL_SIZE_HINT, &nReq); + } + } /* Iterate through the contents of the WAL, copying data to the db file. */ while( rc==SQLITE_OK && 0==walIteratorNext(pIter, &iDbpage, &iFrame) ){ i64 iOffset; assert( walFramePgno(pWal, iFrame)==iDbpage ); - if( iFrame<=nBackfill || iFrame>mxSafeFrame ) continue; + if( iFrame<=nBackfill || iFrame>mxSafeFrame || iDbpage>mxPage ) continue; iOffset = walFrameOffset(iFrame, szPage) + WAL_FRAME_HDRSIZE; /* testcase( IS_BIG_INT(iOffset) ); // requires a 4GiB WAL file */ rc = sqlite3OsRead(pWal->pWalFd, zBuf, szPage, iOffset); if( rc!=SQLITE_OK ) break; iOffset = (iDbpage-1)*(i64)szPage; @@ -41883,11 +42950,11 @@ WalIndexHdr volatile *aHdr; /* Header in shared memory */ /* The first page of the wal-index must be mapped at this point. */ assert( pWal->nWiData>0 && pWal->apWiData[0] ); - /* Read the header. This might happen currently with a write to the + /* Read the header. This might happen concurrently with a write to the ** same area of shared memory on a different CPU in a SMP, ** meaning it is possible that an inconsistent snapshot is read ** from the file. If this happens, return non-zero. ** ** There are two copies of the header at the beginning of the wal-index. @@ -41912,11 +42979,13 @@ } if( memcmp(&pWal->hdr, &h1, sizeof(WalIndexHdr)) ){ *pChanged = 1; memcpy(&pWal->hdr, &h1, sizeof(WalIndexHdr)); - pWal->szPage = pWal->hdr.szPage; + pWal->szPage = (pWal->hdr.szPage&0xfe00) + ((pWal->hdr.szPage&0x0001)<<16); + testcase( pWal->szPage<=32768 ); + testcase( pWal->szPage>=65536 ); } /* The header was successfully read. Return zero. */ return 0; } @@ -42231,10 +43300,11 @@ /* ** Finish with a read transaction. All this does is release the ** read-lock. */ SQLITE_PRIVATE void sqlite3WalEndReadTransaction(Wal *pWal){ + sqlite3WalEndWriteTransaction(pWal); if( pWal->readLock>=0 ){ walUnlockShared(pWal, WAL_READ_LOCK(pWal->readLock)); pWal->readLock = -1; } } @@ -42341,11 +43411,17 @@ /* If iRead is non-zero, then it is the log frame number that contains the ** required page. Read and return data from the log file. */ if( iRead ){ - i64 iOffset = walFrameOffset(iRead, pWal->hdr.szPage) + WAL_FRAME_HDRSIZE; + int sz; + i64 iOffset; + sz = pWal->hdr.szPage; + sz = (pWal->hdr.szPage&0xfe00) + ((pWal->hdr.szPage&0x0001)<<16); + testcase( sz<=32768 ); + testcase( sz>=65536 ); + iOffset = walFrameOffset(iRead, sz) + WAL_FRAME_HDRSIZE; *pInWal = 1; /* testcase( IS_BIG_INT(iOffset) ); // requires a 4GiB WAL */ return sqlite3OsRead(pWal->pWalFd, pOut, nOut, iOffset); } @@ -42353,15 +43429,17 @@ return SQLITE_OK; } /* -** Set *pPgno to the size of the database file (or zero, if unknown). +** Return the size of the database in pages (or zero, if unknown). */ -SQLITE_PRIVATE void sqlite3WalDbsize(Wal *pWal, Pgno *pPgno){ - assert( pWal->readLock>=0 || pWal->lockError ); - *pPgno = pWal->hdr.nPage; +SQLITE_PRIVATE Pgno sqlite3WalDbsize(Wal *pWal){ + if( pWal && ALWAYS(pWal->readLock>=0) ){ + return pWal->hdr.nPage; + } + return 0; } /* ** This function starts a write transaction on the WAL. @@ -42433,11 +43511,11 @@ ** Otherwise, if the callback function does not return an error, this ** function returns SQLITE_OK. */ SQLITE_PRIVATE int sqlite3WalUndo(Wal *pWal, int (*xUndo)(void *, Pgno), void *pUndoCtx){ int rc = SQLITE_OK; - if( pWal->writeLock ){ + if( ALWAYS(pWal->writeLock) ){ Pgno iMax = pWal->hdr.mxFrame; Pgno iFrame; /* Restore the clients cache of the wal-index header to the state it ** was in before the client began writing to the database. @@ -42521,11 +43599,11 @@ ** it sets pWal->hdr.mxFrame to 0. Otherwise, pWal->hdr.mxFrame is left ** unchanged. ** ** SQLITE_OK is returned if no error is encountered (regardless of whether ** or not pWal->hdr.mxFrame is modified). An SQLite error code is returned -** if some error +** if an error occurs. */ static int walRestartLog(Wal *pWal){ int rc = SQLITE_OK; int cnt; @@ -42554,10 +43632,12 @@ walIndexWriteHdr(pWal); pInfo->nBackfill = 0; for(i=1; i<WAL_NREADER; i++) pInfo->aReadMark[i] = READMARK_NOT_USED; assert( pInfo->aReadMark[0]==0 ); walUnlockExclusive(pWal, WAL_READ_LOCK(1), WAL_NREADER-1); + }else if( rc!=SQLITE_BUSY ){ + return rc; } } walUnlockShared(pWal, WAL_READ_LOCK(0)); pWal->readLock = -1; cnt = 0; @@ -42622,11 +43702,11 @@ memcpy(&aWalHdr[16], pWal->hdr.aSalt, 8); walChecksumBytes(1, aWalHdr, WAL_HDRSIZE-2*4, 0, aCksum); sqlite3Put4byte(&aWalHdr[24], aCksum[0]); sqlite3Put4byte(&aWalHdr[28], aCksum[1]); - pWal->szPage = (u16)szPage; + pWal->szPage = szPage; pWal->hdr.bigEndCksum = SQLITE_BIGENDIAN; pWal->hdr.aFrameCksum[0] = aCksum[0]; pWal->hdr.aFrameCksum[1] = aCksum[1]; rc = sqlite3OsWrite(pWal->pWalFd, aWalHdr, sizeof(aWalHdr), 0); @@ -42633,11 +43713,11 @@ WALTRACE(("WAL%p: wal-header write %s\n", pWal, rc ? "failed" : "ok")); if( rc!=SQLITE_OK ){ return rc; } } - assert( pWal->szPage==szPage ); + assert( (int)pWal->szPage==szPage ); /* Write the log file. */ for(p=pList; p; p=p->pDirty){ u32 nDbsize; /* Db-size field for frame header */ i64 iOffset; /* Write offset in log file */ @@ -42717,11 +43797,13 @@ rc = walIndexAppend(pWal, iFrame, pLast->pgno); } if( rc==SQLITE_OK ){ /* Update the private copy of the header. */ - pWal->hdr.szPage = (u16)szPage; + pWal->hdr.szPage = (u16)((szPage&0xff00) | (szPage>>16)); + testcase( szPage<=32768 ); + testcase( szPage>=65536 ); pWal->hdr.mxFrame = iFrame; if( isCommit ){ pWal->hdr.iChange++; pWal->hdr.nPage = nTruncate; } @@ -42929,11 +44011,11 @@ ** ** FORMAT DETAILS ** ** The file is divided into pages. The first page is called page 1, ** the second is page 2, and so forth. A page number of zero indicates -** "no such page". The page size can be any power of 2 between 512 and 32768. +** "no such page". The page size can be any power of 2 between 512 and 65536. ** Each page can be either a btree page, a freelist page, an overflow ** page, or a pointer-map page. ** ** The first page is always a btree page. The first 100 bytes of the first ** page contain a special header (the "file header") that describes the file. @@ -43291,22 +44373,23 @@ MemPage *pPage1; /* First page of the database */ u8 readOnly; /* True if the underlying file is readonly */ u8 pageSizeFixed; /* True if the page size can no longer be changed */ u8 secureDelete; /* True if secure_delete is enabled */ u8 initiallyEmpty; /* Database is empty at start of transaction */ + u8 openFlags; /* Flags to sqlite3BtreeOpen() */ #ifndef SQLITE_OMIT_AUTOVACUUM u8 autoVacuum; /* True if auto-vacuum is enabled */ u8 incrVacuum; /* True if incr-vacuum is enabled */ #endif - u16 pageSize; /* Total number of bytes on a page */ - u16 usableSize; /* Number of usable bytes on each page */ u16 maxLocal; /* Maximum local payload in non-LEAFDATA tables */ u16 minLocal; /* Minimum local payload in non-LEAFDATA tables */ u16 maxLeaf; /* Maximum local payload in a LEAFDATA table */ u16 minLeaf; /* Minimum local payload in a LEAFDATA table */ u8 inTransaction; /* Transaction state */ u8 doNotUseWAL; /* If true, do not open write-ahead-log file */ + u32 pageSize; /* Total number of bytes on a page */ + u32 usableSize; /* Number of usable bytes on each page */ int nTransaction; /* Number of open transactions (read + write) */ u32 nPage; /* Number of pages in the database */ void *pSchema; /* Pointer to space allocated by sqlite3BtreeSchema() */ void (*xFreeSchema)(void*); /* Destructor for BtShared.pSchema */ sqlite3_mutex *mutex; /* Non-recursive mutex required to access this struct */ @@ -43901,11 +44984,20 @@ # define TRACE(X) if(sqlite3BtreeTrace){printf X;fflush(stdout);} #else # define TRACE(X) #endif - +/* +** Extract a 2-byte big-endian integer from an array of unsigned bytes. +** But if the value is zero, make it 65536. +** +** This routine is used to extract the "offset to cell content area" value +** from the header of a btree page. If the page size is 65536 and the page +** is empty, the offset should be 65536, but the 2-byte value stores zero. +** This routine makes the necessary adjustment to 65536. +*/ +#define get2byteNotZero(X) (((((int)get2byte(X))-1)&0xffff)+1) #ifndef SQLITE_OMIT_SHARED_CACHE /* ** A list of BtShared objects that are eligible for participation ** in shared cache. This variable has file scope during normal builds, @@ -44590,15 +45682,20 @@ #ifndef SQLITE_OMIT_AUTOVACUUM /* ** Given a page number of a regular database page, return the page ** number for the pointer-map page that contains the entry for the ** input page number. +** +** Return 0 (not a valid page) for pgno==1 since there is +** no pointer map associated with page 1. The integrity_check logic +** requires that ptrmapPageno(*,1)!=1. */ static Pgno ptrmapPageno(BtShared *pBt, Pgno pgno){ int nPagesPerMapPage; Pgno iPtrMap, ret; assert( sqlite3_mutex_held(pBt->mutex) ); + if( pgno<2 ) return 0; nPagesPerMapPage = (pBt->usableSize/5)+1; iPtrMap = (pgno-2)/nPagesPerMapPage; ret = (iPtrMap*nPagesPerMapPage) + 2; if( ret==PENDING_BYTE_PAGE(pBt) ){ ret++; @@ -45023,21 +46120,21 @@ assert( nByte < usableSize-8 ); nFrag = data[hdr+7]; assert( pPage->cellOffset == hdr + 12 - 4*pPage->leaf ); gap = pPage->cellOffset + 2*pPage->nCell; - top = get2byte(&data[hdr+5]); + top = get2byteNotZero(&data[hdr+5]); if( gap>top ) return SQLITE_CORRUPT_BKPT; testcase( gap+2==top ); testcase( gap+1==top ); testcase( gap==top ); if( nFrag>=60 ){ /* Always defragment highly fragmented pages */ rc = defragmentPage(pPage); if( rc ) return rc; - top = get2byte(&data[hdr+5]); + top = get2byteNotZero(&data[hdr+5]); }else if( gap+2<=top ){ /* Search the freelist looking for a free slot big enough to satisfy ** the request. The allocation is made from the first free slot in ** the list that is large enough to accomadate it. */ @@ -45075,11 +46172,11 @@ */ testcase( gap+2+nByte==top ); if( gap+2+nByte>top ){ rc = defragmentPage(pPage); if( rc ) return rc; - top = get2byte(&data[hdr+5]); + top = get2byteNotZero(&data[hdr+5]); assert( gap+nByte<=top ); } /* Allocate memory from the gap in between the cell pointer array @@ -45241,28 +46338,28 @@ if( !pPage->isInit ){ u16 pc; /* Address of a freeblock within pPage->aData[] */ u8 hdr; /* Offset to beginning of page header */ u8 *data; /* Equal to pPage->aData */ BtShared *pBt; /* The main btree structure */ - u16 usableSize; /* Amount of usable space on each page */ + int usableSize; /* Amount of usable space on each page */ u16 cellOffset; /* Offset from start of page to first cell pointer */ - u16 nFree; /* Number of unused bytes on the page */ - u16 top; /* First byte of the cell content area */ + int nFree; /* Number of unused bytes on the page */ + int top; /* First byte of the cell content area */ int iCellFirst; /* First allowable cell or freeblock offset */ int iCellLast; /* Last possible cell or freeblock offset */ pBt = pPage->pBt; hdr = pPage->hdrOffset; data = pPage->aData; if( decodeFlags(pPage, data[hdr]) ) return SQLITE_CORRUPT_BKPT; - assert( pBt->pageSize>=512 && pBt->pageSize<=32768 ); - pPage->maskPage = pBt->pageSize - 1; + assert( pBt->pageSize>=512 && pBt->pageSize<=65536 ); + pPage->maskPage = (u16)(pBt->pageSize - 1); pPage->nOverflow = 0; usableSize = pBt->usableSize; pPage->cellOffset = cellOffset = hdr + 12 - 4*pPage->leaf; - top = get2byte(&data[hdr+5]); + top = get2byteNotZero(&data[hdr+5]); pPage->nCell = get2byte(&data[hdr+3]); if( pPage->nCell>MX_CELL(pBt) ){ /* To many cells for a single page. The page must be corrupt */ return SQLITE_CORRUPT_BKPT; } @@ -45357,17 +46454,17 @@ data[hdr] = (char)flags; first = hdr + 8 + 4*((flags&PTF_LEAF)==0 ?1:0); memset(&data[hdr+1], 0, 4); data[hdr+7] = 0; put2byte(&data[hdr+5], pBt->usableSize); - pPage->nFree = pBt->usableSize - first; + pPage->nFree = (u16)(pBt->usableSize - first); decodeFlags(pPage, flags); pPage->hdrOffset = hdr; pPage->cellOffset = first; pPage->nOverflow = 0; - assert( pBt->pageSize>=512 && pBt->pageSize<=32768 ); - pPage->maskPage = pBt->pageSize - 1; + assert( pBt->pageSize>=512 && pBt->pageSize<=65536 ); + pPage->maskPage = (u16)(pBt->pageSize - 1); pPage->nCell = 0; pPage->isInit = 1; } @@ -45527,15 +46624,24 @@ /* ** Open a database file. ** ** zFilename is the name of the database file. If zFilename is NULL -** a new database with a random name is created. This randomly named -** database file will be deleted when sqlite3BtreeClose() is called. +** then an ephemeral database is created. The ephemeral database might +** be exclusively in memory, or it might use a disk-based memory cache. +** Either way, the ephemeral database will be automatically deleted +** when sqlite3BtreeClose() is called. +** ** If zFilename is ":memory:" then an in-memory database is created ** that is automatically destroyed when it is closed. ** +** The "flags" parameter is a bitmask that might contain bits +** BTREE_OMIT_JOURNAL and/or BTREE_NO_READLOCK. The BTREE_NO_READLOCK +** bit is also set if the SQLITE_NoReadlock flags is set in db->flags. +** These flags are passed through into sqlite3PagerOpen() and must +** be the same values as PAGER_OMIT_JOURNAL and PAGER_NO_READLOCK. +** ** If the database is already opened in the same database connection ** and we are in shared cache mode, then the open will fail with an ** SQLITE_CONSTRAINT error. We cannot allow two or more BtShared ** objects in the same database connection since doing so will lead ** to problems with locking. @@ -45552,10 +46658,13 @@ Btree *p; /* Handle to return */ sqlite3_mutex *mutexOpen = 0; /* Prevents a race condition. Ticket #3537 */ int rc = SQLITE_OK; /* Result code from this function */ u8 nReserve; /* Byte of unused space on each page */ unsigned char zDbHeader[100]; /* Database header content */ + + /* True if opening an ephemeral, temporary database */ + const int isTempDb = zFilename==0 || zFilename[0]==0; /* Set the variable isMemdb to true for an in-memory database, or ** false for a file-based database. This symbol is only required if ** either of the shared-data or autovacuum features are compiled ** into the library. @@ -45562,17 +46671,34 @@ */ #if !defined(SQLITE_OMIT_SHARED_CACHE) || !defined(SQLITE_OMIT_AUTOVACUUM) #ifdef SQLITE_OMIT_MEMORYDB const int isMemdb = 0; #else - const int isMemdb = zFilename && !strcmp(zFilename, ":memory:"); + const int isMemdb = (zFilename && strcmp(zFilename, ":memory:")==0) + || (isTempDb && sqlite3TempInMemory(db)); #endif #endif assert( db!=0 ); assert( sqlite3_mutex_held(db->mutex) ); + assert( (flags&0xff)==flags ); /* flags fit in 8 bits */ + /* Only a BTREE_SINGLE database can be BTREE_UNORDERED */ + assert( (flags & BTREE_UNORDERED)==0 || (flags & BTREE_SINGLE)!=0 ); + + /* A BTREE_SINGLE database is always a temporary and/or ephemeral */ + assert( (flags & BTREE_SINGLE)==0 || isTempDb ); + + if( db->flags & SQLITE_NoReadlock ){ + flags |= BTREE_NO_READLOCK; + } + if( isMemdb ){ + flags |= BTREE_MEMORY; + } + if( (vfsFlags & SQLITE_OPEN_MAIN_DB)!=0 && (isMemdb || isTempDb) ){ + vfsFlags = (vfsFlags & ~SQLITE_OPEN_MAIN_DB) | SQLITE_OPEN_TEMP_DB; + } pVfs = db->pVfs; p = sqlite3MallocZero(sizeof(Btree)); if( !p ){ return SQLITE_NOMEM; } @@ -45586,11 +46712,11 @@ #if !defined(SQLITE_OMIT_SHARED_CACHE) && !defined(SQLITE_OMIT_DISKIO) /* ** If this Btree is a candidate for shared cache, try to find an ** existing BtShared object that we can share with */ - if( isMemdb==0 && zFilename && zFilename[0] ){ + if( isMemdb==0 && isTempDb==0 ){ if( vfsFlags & SQLITE_OPEN_SHAREDCACHE ){ int nFullPathname = pVfs->mxPathname+1; char *zFullPathname = sqlite3Malloc(nFullPathname); sqlite3_mutex *mutexShared; p->sharable = 1; @@ -45661,10 +46787,11 @@ rc = sqlite3PagerReadFileheader(pBt->pPager,sizeof(zDbHeader),zDbHeader); } if( rc!=SQLITE_OK ){ goto btree_open_out; } + pBt->openFlags = (u8)flags; pBt->db = db; sqlite3PagerSetBusyhandler(pBt->pPager, btreeInvokeBusyHandler, pBt); p->pBt = pBt; pBt->pCursor = 0; @@ -45671,11 +46798,11 @@ pBt->pPage1 = 0; pBt->readOnly = sqlite3PagerIsreadonly(pBt->pPager); #ifdef SQLITE_SECURE_DELETE pBt->secureDelete = 1; #endif - pBt->pageSize = get2byte(&zDbHeader[16]); + pBt->pageSize = (zDbHeader[16]<<8) | (zDbHeader[17]<<16); if( pBt->pageSize<512 || pBt->pageSize>SQLITE_MAX_PAGE_SIZE || ((pBt->pageSize-1)&pBt->pageSize)!=0 ){ pBt->pageSize = 0; #ifndef SQLITE_OMIT_AUTOVACUUM /* If the magic name ":memory:" will create an in-memory database, then @@ -45765,10 +46892,18 @@ sqlite3PagerClose(pBt->pPager); } sqlite3_free(pBt); sqlite3_free(p); *ppBtree = 0; + }else{ + /* If the B-Tree was successfully opened, set the pager-cache size to the + ** default value. Except, when opening on an existing shared pager-cache, + ** do not change the pager-cache size. + */ + if( sqlite3BtreeSchema(p, 0, 0)==0 ){ + sqlite3PagerSetCachesize(p->pBt->pPager, SQLITE_DEFAULT_CACHE_SIZE); + } } if( mutexOpen ){ assert( sqlite3_mutex_held(mutexOpen) ); sqlite3_mutex_leave(mutexOpen); } @@ -45985,11 +47120,11 @@ assert( nReserve>=0 && nReserve<=255 ); if( pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE && ((pageSize-1)&pageSize)==0 ){ assert( (pageSize & 7)==0 ); assert( !pBt->pPage1 && !pBt->pCursor ); - pBt->pageSize = (u16)pageSize; + pBt->pageSize = (u32)pageSize; freeTempSpace(pBt); } rc = sqlite3PagerSetPagesize(pBt->pPager, &pBt->pageSize, nReserve); pBt->usableSize = pBt->pageSize - (u16)nReserve; if( iFix ) pBt->pageSizeFixed = 1; @@ -46120,19 +47255,17 @@ /* Do some checking to help insure the file we opened really is ** a valid database file. */ nPage = nPageHeader = get4byte(28+(u8*)pPage1->aData); - if( (rc = sqlite3PagerPagecount(pBt->pPager, &nPageFile))!=SQLITE_OK ){; - goto page1_init_failed; - } + sqlite3PagerPagecount(pBt->pPager, &nPageFile); if( nPage==0 || memcmp(24+(u8*)pPage1->aData, 92+(u8*)pPage1->aData,4)!=0 ){ nPage = nPageFile; } if( nPage>0 ){ - int pageSize; - int usableSize; + u32 pageSize; + u32 usableSize; u8 *page1 = pPage1->aData; rc = SQLITE_NOTADB; if( memcmp(page1, zMagicHeader, 16)!=0 ){ goto page1_init_failed; } @@ -46179,28 +47312,29 @@ ** version 3.6.0, we require them to be fixed. */ if( memcmp(&page1[21], "\100\040\040",3)!=0 ){ goto page1_init_failed; } - pageSize = get2byte(&page1[16]); - if( ((pageSize-1)&pageSize)!=0 || pageSize<512 || - (SQLITE_MAX_PAGE_SIZE<32768 && pageSize>SQLITE_MAX_PAGE_SIZE) + pageSize = (page1[16]<<8) | (page1[17]<<16); + if( ((pageSize-1)&pageSize)!=0 + || pageSize>SQLITE_MAX_PAGE_SIZE + || pageSize<=256 ){ goto page1_init_failed; } assert( (pageSize & 7)==0 ); usableSize = pageSize - page1[20]; - if( pageSize!=pBt->pageSize ){ + if( (u32)pageSize!=pBt->pageSize ){ /* After reading the first page of the database assuming a page size ** of BtShared.pageSize, we have discovered that the page-size is ** actually pageSize. Unlock the database, leave pBt->pPage1 at ** zero and return SQLITE_OK. The caller will call this function ** again with the correct page-size. */ releasePage(pPage1); - pBt->usableSize = (u16)usableSize; - pBt->pageSize = (u16)pageSize; + pBt->usableSize = usableSize; + pBt->pageSize = pageSize; freeTempSpace(pBt); rc = sqlite3PagerSetPagesize(pBt->pPager, &pBt->pageSize, pageSize-usableSize); return rc; } @@ -46209,12 +47343,12 @@ goto page1_init_failed; } if( usableSize<480 ){ goto page1_init_failed; } - pBt->pageSize = (u16)pageSize; - pBt->usableSize = (u16)usableSize; + pBt->pageSize = pageSize; + pBt->usableSize = usableSize; #ifndef SQLITE_OMIT_AUTOVACUUM pBt->autoVacuum = (get4byte(&page1[36 + 4*4])?1:0); pBt->incrVacuum = (get4byte(&page1[36 + 7*4])?1:0); #endif } @@ -46226,18 +47360,18 @@ ** 2-byte pointer to the cell ** 4-byte child pointer ** 9-byte nKey value ** 4-byte nData value ** 4-byte overflow page pointer - ** So a cell consists of a 2-byte poiner, a header which is as much as + ** So a cell consists of a 2-byte pointer, a header which is as much as ** 17 bytes long, 0 to N bytes of payload, and an optional 4 byte overflow ** page pointer. */ - pBt->maxLocal = (pBt->usableSize-12)*64/255 - 23; - pBt->minLocal = (pBt->usableSize-12)*32/255 - 23; - pBt->maxLeaf = pBt->usableSize - 35; - pBt->minLeaf = (pBt->usableSize-12)*32/255 - 23; + pBt->maxLocal = (u16)((pBt->usableSize-12)*64/255 - 23); + pBt->minLocal = (u16)((pBt->usableSize-12)*32/255 - 23); + pBt->maxLeaf = (u16)(pBt->usableSize - 35); + pBt->minLeaf = (u16)((pBt->usableSize-12)*32/255 - 23); assert( pBt->maxLeaf + 23 <= MX_CELL_SIZE(pBt) ); pBt->pPage1 = pPage1; pBt->nPage = nPage; return SQLITE_OK; @@ -46286,11 +47420,12 @@ data = pP1->aData; rc = sqlite3PagerWrite(pP1->pDbPage); if( rc ) return rc; memcpy(data, zMagicHeader, sizeof(zMagicHeader)); assert( sizeof(zMagicHeader)==16 ); - put2byte(&data[16], pBt->pageSize); + data[16] = (u8)((pBt->pageSize>>8)&0xff); + data[17] = (u8)((pBt->pageSize>>16)&0xff); data[18] = 1; data[19] = 1; assert( pBt->usableSize<=pBt->pageSize && pBt->usableSize+255>=pBt->pageSize); data[20] = (u8)(pBt->pageSize - pBt->usableSize); data[21] = 64; @@ -48297,13 +49432,13 @@ c = +1; } pCur->validNKey = 1; pCur->info.nKey = nCellKey; }else{ - /* The maximum supported page-size is 32768 bytes. This means that + /* The maximum supported page-size is 65536 bytes. This means that ** the maximum number of record bytes stored on an index B-Tree - ** page is at most 8198 bytes, which may be stored as a 2-byte + ** page is less than 16384 bytes and may be stored as a 2-byte ** varint. This information is used to attempt to avoid parsing ** the entire cell by checking for the cases where the record is ** stored entirely within the b-tree page by inspecting the first ** 2 bytes of the cell. */ @@ -48662,10 +49797,14 @@ } if( k==0 ){ if( !pPrevTrunk ){ memcpy(&pPage1->aData[32], &pTrunk->aData[0], 4); }else{ + rc = sqlite3PagerWrite(pPrevTrunk->pDbPage); + if( rc!=SQLITE_OK ){ + goto end_allocate_page; + } memcpy(&pPrevTrunk->aData[0], &pTrunk->aData[0], 4); } }else{ /* The trunk page is required by the caller but it contains ** pointers to free-list leaves. The first leaf becomes a trunk @@ -48968,11 +50107,11 @@ BtShared *pBt = pPage->pBt; CellInfo info; Pgno ovflPgno; int rc; int nOvfl; - u16 ovflPageSize; + u32 ovflPageSize; assert( sqlite3_mutex_held(pPage->pBt->mutex) ); btreeParseCellPtr(pPage, pCell, &info); if( info.iOverflow==0 ){ return SQLITE_OK; /* No overflow pages. Return without doing anything */ @@ -49193,11 +50332,11 @@ ** ** "sz" must be the number of bytes in the cell. */ static void dropCell(MemPage *pPage, int idx, int sz, int *pRC){ int i; /* Loop counter */ - int pc; /* Offset to cell content of cell being deleted */ + u32 pc; /* Offset to cell content of cell being deleted */ u8 *data; /* pPage->aData */ u8 *ptr; /* Used to move bytes around within data[] */ int rc; /* The return code */ int hdr; /* Beginning of the header. 0 most pages. 100 page 1 */ @@ -49211,11 +50350,11 @@ ptr = &data[pPage->cellOffset + 2*idx]; pc = get2byte(ptr); hdr = pPage->hdrOffset; testcase( pc==get2byte(&data[hdr+5]) ); testcase( pc+sz==pPage->pBt->usableSize ); - if( pc < get2byte(&data[hdr+5]) || pc+sz > pPage->pBt->usableSize ){ + if( pc < (u32)get2byte(&data[hdr+5]) || pc+sz > pPage->pBt->usableSize ){ *pRC = SQLITE_CORRUPT_BKPT; return; } rc = freeSpace(pPage, pc, sz); if( rc ){ @@ -49268,11 +50407,11 @@ int nSkip = (iChild ? 4 : 0); if( *pRC ) return; assert( i>=0 && i<=pPage->nCell+pPage->nOverflow ); - assert( pPage->nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=5460 ); + assert( pPage->nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=10921 ); assert( pPage->nOverflow<=ArraySize(pPage->aOvfl) ); assert( sqlite3_mutex_held(pPage->pBt->mutex) ); /* The cell should normally be sized correctly. However, when moving a ** malformed cell from a leaf page to an interior page, if the cell size ** wanted to be less than 4 but got rounded up to 4 on the leaf, then size @@ -49348,16 +50487,16 @@ const int hdr = pPage->hdrOffset; /* Offset of header on pPage */ const int nUsable = pPage->pBt->usableSize; /* Usable size of page */ assert( pPage->nOverflow==0 ); assert( sqlite3_mutex_held(pPage->pBt->mutex) ); - assert( nCell>=0 && nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=5460 ); + assert( nCell>=0 && nCell<=MX_CELL(pPage->pBt) && MX_CELL(pPage->pBt)<=10921); assert( sqlite3PagerIswriteable(pPage->pDbPage) ); /* Check that the page has just been zeroed by zeroPage() */ assert( pPage->nCell==0 ); - assert( get2byte(&data[hdr+5])==nUsable ); + assert( get2byteNotZero(&data[hdr+5])==nUsable ); pCellptr = &data[pPage->cellOffset + nCell*2]; cellbody = nUsable; for(i=nCell-1; i>=0; i--){ pCellptr -= 2; @@ -49419,10 +50558,11 @@ assert( sqlite3_mutex_held(pPage->pBt->mutex) ); assert( sqlite3PagerIswriteable(pParent->pDbPage) ); assert( pPage->nOverflow==1 ); + /* This error condition is now caught prior to reaching this function */ if( pPage->nCell<=0 ) return SQLITE_CORRUPT_BKPT; /* Allocate a new page. This page will become the right-sibling of ** pPage. Make the parent page writable, so that the new divider cell ** may be inserted. If both these operations are successful, proceed. @@ -49748,11 +50888,11 @@ ** In this case, temporarily copy the cell into the aOvflSpace[] ** buffer. It will be copied out again as soon as the aSpace[] buffer ** is allocated. */ if( pBt->secureDelete ){ int iOff = SQLITE_PTR_TO_INT(apDiv[i]) - SQLITE_PTR_TO_INT(pParent->aData); - if( (iOff+szNew[i])>pBt->usableSize ){ + if( (iOff+szNew[i])>(int)pBt->usableSize ){ rc = SQLITE_CORRUPT_BKPT; memset(apOld, 0, (i+1)*sizeof(MemPage*)); goto balance_cleanup; }else{ memcpy(&aOvflSpace[iOff], apDiv[i], szNew[i]); @@ -49827,11 +50967,11 @@ u8 *pTemp; assert( nCell<nMaxCells ); szCell[nCell] = sz; pTemp = &aSpace1[iSpace1]; iSpace1 += sz; - assert( sz<=pBt->pageSize/4 ); + assert( sz<=pBt->maxLocal+23 ); assert( iSpace1<=pBt->pageSize ); memcpy(pTemp, apDiv[i], sz); apCell[nCell] = pTemp+leafCorrection; assert( leafCorrection==0 || leafCorrection==4 ); szCell[nCell] = szCell[nCell] - leafCorrection; @@ -50073,11 +51213,11 @@ assert(leafCorrection==4); sz = cellSizePtr(pParent, pCell); } } iOvflSpace += sz; - assert( sz<=pBt->pageSize/4 ); + assert( sz<=pBt->maxLocal+23 ); assert( iOvflSpace<=pBt->pageSize ); insertCell(pParent, nxDiv, pCell, sz, pTemp, pNew->pgno, &rc); if( rc!=SQLITE_OK ) goto balance_cleanup; assert( sqlite3PagerIswriteable(pParent->pDbPage) ); @@ -50710,15 +51850,16 @@ ** flags might not work: ** ** BTREE_INTKEY|BTREE_LEAFDATA Used for SQL tables with rowid keys ** BTREE_ZERODATA Used for SQL indices */ -static int btreeCreateTable(Btree *p, int *piTable, int flags){ +static int btreeCreateTable(Btree *p, int *piTable, int createTabFlags){ BtShared *pBt = p->pBt; MemPage *pRoot; Pgno pgnoRoot; int rc; + int ptfFlags; /* Page-type flage for the root page of new table */ assert( sqlite3BtreeHoldsMutex(p) ); assert( pBt->inTransaction==TRANS_WRITE ); assert( !pBt->readOnly ); @@ -50833,12 +51974,18 @@ rc = allocateBtreePage(pBt, &pRoot, &pgnoRoot, 1, 0); if( rc ) return rc; } #endif assert( sqlite3PagerIswriteable(pRoot->pDbPage) ); - zeroPage(pRoot, flags | PTF_LEAF); + if( createTabFlags & BTREE_INTKEY ){ + ptfFlags = PTF_INTKEY | PTF_LEAFDATA | PTF_LEAF; + }else{ + ptfFlags = PTF_ZERODATA | PTF_LEAF; + } + zeroPage(pRoot, ptfFlags); sqlite3PagerUnref(pRoot->pDbPage); + assert( (pBt->openFlags & BTREE_SINGLE)==0 || pgnoRoot==2 ); *piTable = (int)pgnoRoot; return SQLITE_OK; } SQLITE_PRIVATE int sqlite3BtreeCreateTable(Btree *p, int *piTable, int flags){ int rc; @@ -51317,11 +52464,11 @@ #ifndef SQLITE_OMIT_AUTOVACUUM if( pCheck->pBt->autoVacuum ){ checkPtrmap(pCheck, iPage, PTRMAP_FREEPAGE, 0, zContext); } #endif - if( n>pCheck->pBt->usableSize/4-2 ){ + if( n>(int)pCheck->pBt->usableSize/4-2 ){ checkAppendMsg(pCheck, zContext, "freelist leaf count too big on page %d", iPage); N--; }else{ for(i=0; i<n; i++){ @@ -51528,24 +52675,24 @@ hdr = pPage->hdrOffset; hit = sqlite3PageMalloc( pBt->pageSize ); if( hit==0 ){ pCheck->mallocFailed = 1; }else{ - u16 contentOffset = get2byte(&data[hdr+5]); + int contentOffset = get2byteNotZero(&data[hdr+5]); assert( contentOffset<=usableSize ); /* Enforced by btreeInitPage() */ memset(hit+contentOffset, 0, usableSize-contentOffset); memset(hit, 1, contentOffset); nCell = get2byte(&data[hdr+3]); cellStart = hdr + 12 - 4*pPage->leaf; for(i=0; i<nCell; i++){ int pc = get2byte(&data[cellStart+i*2]); - u16 size = 1024; + u32 size = 65536; int j; if( pc<=usableSize-4 ){ size = cellSizePtr(pPage, &data[pc]); } - if( (pc+size-1)>=usableSize ){ + if( (int)(pc+size-1)>=usableSize ){ checkAppendMsg(pCheck, 0, "Corruption detected in cell %d on page %d",i,iPage); }else{ for(j=pc+size-1; j>=pc; j--) hit[j]++; } @@ -52094,11 +53241,14 @@ sqlite3Error( pDestDb, SQLITE_ERROR, "source and destination must be distinct" ); p = 0; }else { - /* Allocate space for a new sqlite3_backup object */ + /* Allocate space for a new sqlite3_backup object... + ** EVIDENCE-OF: R-64852-21591 The sqlite3_backup object is created by a + ** call to sqlite3_backup_init() and is destroyed by a call to + ** sqlite3_backup_finish(). */ p = (sqlite3_backup *)sqlite3_malloc(sizeof(sqlite3_backup)); if( !p ){ sqlite3Error(pDestDb, SQLITE_NOMEM, 0); } } @@ -52164,10 +53314,19 @@ ** page sizes of the source and destination differ. */ if( nSrcPgsz!=nDestPgsz && sqlite3PagerIsMemdb(pDestPager) ){ rc = SQLITE_READONLY; } + +#ifdef SQLITE_HAS_CODEC + /* Backup is not possible if the page size of the destination is changing + ** a a codec is in use. + */ + if( nSrcPgsz!=nDestPgsz && sqlite3PagerGetCodec(pDestPager)!=0 ){ + rc = SQLITE_READONLY; + } +#endif /* This loop runs once for each destination page spanned by the source ** page. For each iteration, variable iOff is set to the byte offset ** of the destination page. */ @@ -52468,10 +53627,13 @@ if( p->pDestDb ){ sqlite3_mutex_leave(p->pDestDb->mutex); } sqlite3BtreeLeave(p->pSrc); if( p->pDestDb ){ + /* EVIDENCE-OF: R-64852-21591 The sqlite3_backup object is created by a + ** call to sqlite3_backup_init() and is destroyed by a call to + ** sqlite3_backup_finish(). */ sqlite3_free(p); } sqlite3_mutex_leave(mutex); return rc; } @@ -52719,10 +53881,13 @@ return SQLITE_NOMEM; } pMem->z[pMem->n] = 0; pMem->z[pMem->n+1] = 0; pMem->flags |= MEM_Term; +#ifdef SQLITE_DEBUG + pMem->pScopyFrom = 0; +#endif } return SQLITE_OK; } @@ -52839,11 +54004,11 @@ memset(&ctx, 0, sizeof(ctx)); ctx.s.flags = MEM_Null; ctx.s.db = pMem->db; ctx.pMem = pMem; ctx.pFunc = pFunc; - pFunc->xFinalize(&ctx); + pFunc->xFinalize(&ctx); /* IMP: R-24505-23230 */ assert( 0==(pMem->flags&MEM_Dyn) && !pMem->xDel ); sqlite3DbFree(pMem->db, pMem->zMalloc); memcpy(pMem, &ctx.s, sizeof(ctx.s)); rc = ctx.isError; } @@ -52952,17 +54117,13 @@ return pMem->u.i; }else if( flags & MEM_Real ){ return doubleToInt64(pMem->r); }else if( flags & (MEM_Str|MEM_Blob) ){ i64 value; - pMem->flags |= MEM_Str; - if( sqlite3VdbeChangeEncoding(pMem, SQLITE_UTF8) - || sqlite3VdbeMemNulTerminate(pMem) ){ - return 0; - } - assert( pMem->z ); - sqlite3Atoi64(pMem->z, &value); + assert( pMem->z || pMem->n==0 ); + testcase( pMem->z==0 ); + sqlite3Atoi64(pMem->z, &value, pMem->n, pMem->enc); return value; }else{ return 0; } } @@ -52981,18 +54142,11 @@ }else if( pMem->flags & MEM_Int ){ return (double)pMem->u.i; }else if( pMem->flags & (MEM_Str|MEM_Blob) ){ /* (double)0 In case of SQLITE_OMIT_FLOATING_POINT... */ double val = (double)0; - pMem->flags |= MEM_Str; - if( sqlite3VdbeChangeEncoding(pMem, SQLITE_UTF8) - || sqlite3VdbeMemNulTerminate(pMem) ){ - /* (double)0 In case of SQLITE_OMIT_FLOATING_POINT... */ - return (double)0; - } - assert( pMem->z ); - sqlite3AtoF(pMem->z, &val); + sqlite3AtoF(pMem->z, &val, pMem->n, pMem->enc); return val; }else{ /* (double)0 In case of SQLITE_OMIT_FLOATING_POINT... */ return (double)0; } @@ -53061,25 +54215,23 @@ ** Every effort is made to force the conversion, even if the input ** is a string that does not look completely like a number. Convert ** as much of the string as we can and ignore the rest. */ SQLITE_PRIVATE int sqlite3VdbeMemNumerify(Mem *pMem){ - int rc; - assert( (pMem->flags & (MEM_Int|MEM_Real|MEM_Null))==0 ); - assert( (pMem->flags & (MEM_Blob|MEM_Str))!=0 ); - assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) ); - rc = sqlite3VdbeChangeEncoding(pMem, SQLITE_UTF8); - if( rc ) return rc; - rc = sqlite3VdbeMemNulTerminate(pMem); - if( rc ) return rc; - if( sqlite3Atoi64(pMem->z, &pMem->u.i) ){ - MemSetTypeFlag(pMem, MEM_Int); - }else{ - pMem->r = sqlite3VdbeRealValue(pMem); - MemSetTypeFlag(pMem, MEM_Real); - sqlite3VdbeIntegerAffinity(pMem); - } + if( (pMem->flags & (MEM_Int|MEM_Real|MEM_Null))==0 ){ + assert( (pMem->flags & (MEM_Blob|MEM_Str))!=0 ); + assert( pMem->db==0 || sqlite3_mutex_held(pMem->db->mutex) ); + if( 0==sqlite3Atoi64(pMem->z, &pMem->u.i, pMem->n, pMem->enc) ){ + MemSetTypeFlag(pMem, MEM_Int); + }else{ + pMem->r = sqlite3VdbeRealValue(pMem); + MemSetTypeFlag(pMem, MEM_Real); + sqlite3VdbeIntegerAffinity(pMem); + } + } + assert( (pMem->flags & (MEM_Int|MEM_Real|MEM_Null))!=0 ); + pMem->flags &= ~(MEM_Str|MEM_Blob); return SQLITE_OK; } /* ** Delete any previous value and set the value stored in *pMem to NULL. @@ -53180,10 +54332,32 @@ return n>p->db->aLimit[SQLITE_LIMIT_LENGTH]; } return 0; } +#ifdef SQLITE_DEBUG +/* +** This routine prepares a memory cell for modication by breaking +** its link to a shallow copy and by marking any current shallow +** copies of this cell as invalid. +** +** This is used for testing and debugging only - to make sure shallow +** copies are not misused. +*/ +SQLITE_PRIVATE void sqlite3VdbeMemPrepareToChange(Vdbe *pVdbe, Mem *pMem){ + int i; + Mem *pX; + for(i=1, pX=&pVdbe->aMem[1]; i<=pVdbe->nMem; i++, pX++){ + if( pX->pScopyFrom==pMem ){ + pX->flags |= MEM_Invalid; + pX->pScopyFrom = 0; + } + } + pMem->pScopyFrom = 0; +} +#endif /* SQLITE_DEBUG */ + /* ** Size of struct Mem not including the Mem.zMalloc member. */ #define MEMCELLSIZE (size_t)(&(((Mem *)0)->zMalloc)) @@ -53548,11 +54722,11 @@ assert( (pVal->flags & (MEM_Ephem|MEM_Static))!=0 ); if( sqlite3VdbeMemMakeWriteable(pVal)!=SQLITE_OK ){ return 0; } } - sqlite3VdbeMemNulTerminate(pVal); + sqlite3VdbeMemNulTerminate(pVal); /* IMP: R-59893-45467 */ }else{ assert( (pVal->flags&MEM_Blob)==0 ); sqlite3VdbeMemStringify(pVal, enc); assert( 0==(1&SQLITE_PTR_TO_INT(pVal->z)) ); } @@ -53596,10 +54770,12 @@ sqlite3_value **ppVal /* Write the new value here */ ){ int op; char *zVal = 0; sqlite3_value *pVal = 0; + int negInt = 1; + const char *zNeg = ""; if( !pExpr ){ *ppVal = 0; return SQLITE_OK; } @@ -53612,35 +54788,50 @@ #ifdef SQLITE_ENABLE_STAT2 if( op==TK_REGISTER ) op = pExpr->op2; #else if( NEVER(op==TK_REGISTER) ) op = pExpr->op2; #endif + + /* Handle negative integers in a single step. This is needed in the + ** case when the value is -9223372036854775808. + */ + if( op==TK_UMINUS + && (pExpr->pLeft->op==TK_INTEGER || pExpr->pLeft->op==TK_FLOAT) ){ + pExpr = pExpr->pLeft; + op = pExpr->op; + negInt = -1; + zNeg = "-"; + } if( op==TK_STRING || op==TK_FLOAT || op==TK_INTEGER ){ pVal = sqlite3ValueNew(db); if( pVal==0 ) goto no_mem; if( ExprHasProperty(pExpr, EP_IntValue) ){ - sqlite3VdbeMemSetInt64(pVal, (i64)pExpr->u.iValue); + sqlite3VdbeMemSetInt64(pVal, (i64)pExpr->u.iValue*negInt); }else{ - zVal = sqlite3DbStrDup(db, pExpr->u.zToken); + zVal = sqlite3MPrintf(db, "%s%s", zNeg, pExpr->u.zToken); if( zVal==0 ) goto no_mem; sqlite3ValueSetStr(pVal, -1, zVal, SQLITE_UTF8, SQLITE_DYNAMIC); if( op==TK_FLOAT ) pVal->type = SQLITE_FLOAT; } if( (op==TK_INTEGER || op==TK_FLOAT ) && affinity==SQLITE_AFF_NONE ){ sqlite3ValueApplyAffinity(pVal, SQLITE_AFF_NUMERIC, SQLITE_UTF8); }else{ sqlite3ValueApplyAffinity(pVal, affinity, SQLITE_UTF8); } + if( pVal->flags & (MEM_Int|MEM_Real) ) pVal->flags &= ~MEM_Str; if( enc!=SQLITE_UTF8 ){ sqlite3VdbeChangeEncoding(pVal, enc); } }else if( op==TK_UMINUS ) { + /* This branch happens for multiple negative signs. Ex: -(-5) */ if( SQLITE_OK==sqlite3ValueFromExpr(db,pExpr->pLeft,enc,affinity,&pVal) ){ + sqlite3VdbeMemNumerify(pVal); pVal->u.i = -1 * pVal->u.i; /* (double)-1 In case of SQLITE_OMIT_FLOATING_POINT... */ pVal->r = (double)-1 * pVal->r; + sqlite3ValueApplyAffinity(pVal, affinity, enc); } } #ifndef SQLITE_OMIT_BLOB_LITERAL else if( op==TK_BLOB ){ int nVal; @@ -55463,13 +56654,14 @@ */ for(i=0; i<db->nDb; i++){ Btree *pBt = db->aDb[i].pBt; if( sqlite3BtreeIsInTrans(pBt) ){ char const *zFile = sqlite3BtreeGetJournalname(pBt); - if( zFile==0 || zFile[0]==0 ){ + if( zFile==0 ){ continue; /* Ignore TEMP and :memory: databases */ } + assert( zFile[0]!=0 ); if( !needSync && !sqlite3BtreeSyncDisabled(pBt) ){ needSync = 1; } rc = sqlite3OsWrite(pMaster, zFile, sqlite3Strlen30(zFile)+1, offset); offset += sqlite3Strlen30(zFile)+1; @@ -55769,12 +56961,21 @@ mrc = p->rc & 0xff; assert( p->rc!=SQLITE_IOERR_BLOCKED ); /* This error no longer exists */ isSpecialError = mrc==SQLITE_NOMEM || mrc==SQLITE_IOERR || mrc==SQLITE_INTERRUPT || mrc==SQLITE_FULL; if( isSpecialError ){ - /* If the query was read-only, we need do no rollback at all. Otherwise, - ** proceed with the special handling. + /* If the query was read-only and the error code is SQLITE_INTERRUPT, + ** no rollback is necessary. Otherwise, at least a savepoint + ** transaction must be rolled back to restore the database to a + ** consistent state. + ** + ** Even if the statement is read-only, it is important to perform + ** a statement or transaction rollback operation. If the error + ** occured while writing to the journal, sub-journal or database + ** file as part of an effort to free up cache space (see function + ** pagerStress() in pager.c), the rollback is required to restore + ** the pager to a consistent state. */ if( !p->readOnly || mrc!=SQLITE_INTERRUPT ){ if( (mrc==SQLITE_NOMEM || mrc==SQLITE_FULL) && p->usesStmtJournal ){ eStatementOp = SAVEPOINT_ROLLBACK; }else{ @@ -56920,10 +58121,12 @@ ** sqlite3_errcode(), sqlite3_errmsg() and sqlite3_errmsg16(). */ SQLITE_API int sqlite3_finalize(sqlite3_stmt *pStmt){ int rc; if( pStmt==0 ){ + /* IMPLEMENTATION-OF: R-57228-12904 Invoking sqlite3_finalize() on a NULL + ** pointer is a harmless no-op. */ rc = SQLITE_OK; }else{ Vdbe *v = (Vdbe*)pStmt; sqlite3 *db = v->db; #if SQLITE_THREADSAFE @@ -56996,11 +58199,11 @@ Mem *p = (Mem*)pVal; if( p->flags & (MEM_Blob|MEM_Str) ){ sqlite3VdbeMemExpandBlob(p); p->flags &= ~MEM_Str; p->flags |= MEM_Blob; - return p->z; + return p->n ? p->z : 0; }else{ return sqlite3_value_text(pVal); } } SQLITE_API int sqlite3_value_bytes(sqlite3_value *pVal){ @@ -57350,10 +58553,16 @@ } /* ** Extract the user data from a sqlite3_context structure and return a ** pointer to it. +** +** IMPLEMENTATION-OF: R-46798-50301 The sqlite3_context_db_handle() interface +** returns a copy of the pointer to the database connection (the 1st +** parameter) of the sqlite3_create_function() and +** sqlite3_create_function16() routines that originally registered the +** application defined function. */ SQLITE_API sqlite3 *sqlite3_context_db_handle(sqlite3_context *p){ assert( p && p->pFunc ); return p->s.db; } @@ -57559,12 +58768,11 @@ ** sqlite3_column_text() ** sqlite3_column_text16() ** sqlite3_column_real() ** sqlite3_column_bytes() ** sqlite3_column_bytes16() -** -** But not for sqlite3_column_blob(), which never calls malloc(). +** sqiite3_column_blob() */ static void columnMallocFailure(sqlite3_stmt *pStmt) { /* If malloc() failed during an encoding conversion within an ** sqlite3_column_XXX API, then set the return code of the statement to @@ -57828,10 +59036,16 @@ pVar->flags = MEM_Null; sqlite3Error(p->db, SQLITE_OK, 0); /* If the bit corresponding to this variable in Vdbe.expmask is set, then ** binding a new value to this variable invalidates the current query plan. + ** + ** IMPLEMENTATION-OF: R-48440-37595 If the specific value bound to host + ** parameter in the WHERE clause might influence the choice of query plan + ** for a statement, then the statement will be automatically recompiled, + ** as if there had been a schema change, on the first sqlite3_step() call + ** following any change to the bindings of that parameter. */ if( p->isPrepareV2 && ((i<32 && p->expmask & ((u32)1 << i)) || p->expmask==0xffffffff) ){ p->expired = 1; @@ -58325,10 +59539,21 @@ ** of the code in this file is, therefore, important. See other comments ** in this file for details. If in doubt, do not deviate from existing ** commenting and indentation practices when changing or adding code. */ +/* +** Invoke this macro on memory cells just prior to changing the +** value of the cell. This macro verifies that shallow copies are +** not misused. +*/ +#ifdef SQLITE_DEBUG +# define memAboutToChange(P,M) sqlite3VdbeMemPrepareToChange(P,M) +#else +# define memAboutToChange(P,M) +#endif + /* ** The following global variable is incremented every time a cursor ** moves, either by the OP_SeekXX, OP_Next, or OP_Prev opcodes. The test ** procedures use this information to make sure that indices are ** working correctly. This variable has no function other than to @@ -58517,35 +59742,21 @@ ** looks like a number, convert it into a number. If it does not ** look like a number, leave it alone. */ static void applyNumericAffinity(Mem *pRec){ if( (pRec->flags & (MEM_Real|MEM_Int))==0 ){ - int realnum; + double rValue; + i64 iValue; u8 enc = pRec->enc; - sqlite3VdbeMemNulTerminate(pRec); - if( (pRec->flags&MEM_Str) && sqlite3IsNumber(pRec->z, &realnum, enc) ){ - i64 value; - char *zUtf8 = pRec->z; -#ifndef SQLITE_OMIT_UTF16 - if( enc!=SQLITE_UTF8 ){ - assert( pRec->db ); - zUtf8 = sqlite3Utf16to8(pRec->db, pRec->z, pRec->n, enc); - if( !zUtf8 ) return; - } -#endif - if( !realnum && sqlite3Atoi64(zUtf8, &value) ){ - pRec->u.i = value; - MemSetTypeFlag(pRec, MEM_Int); - }else{ - sqlite3AtoF(zUtf8, &pRec->r); - MemSetTypeFlag(pRec, MEM_Real); - } -#ifndef SQLITE_OMIT_UTF16 - if( enc!=SQLITE_UTF8 ){ - sqlite3DbFree(pRec->db, zUtf8); - } -#endif + if( (pRec->flags&MEM_Str)==0 ) return; + if( sqlite3AtoF(pRec->z, &rValue, pRec->n, enc)==0 ) return; + if( 0==sqlite3Atoi64(pRec->z, &iValue, pRec->n, enc) ){ + pRec->u.i = iValue; + pRec->flags |= MEM_Int; + }else{ + pRec->r = rValue; + pRec->flags |= MEM_Real; } } } /* @@ -59434,38 +60645,44 @@ assert( pOp->opflags==sqlite3OpcodeProperty[pOp->opcode] ); if( pOp->opflags & OPFLG_OUT2_PRERELEASE ){ assert( pOp->p2>0 ); assert( pOp->p2<=p->nMem ); pOut = &aMem[pOp->p2]; + memAboutToChange(p, pOut); sqlite3VdbeMemReleaseExternal(pOut); pOut->flags = MEM_Int; } /* Sanity checking on other operands */ #ifdef SQLITE_DEBUG if( (pOp->opflags & OPFLG_IN1)!=0 ){ assert( pOp->p1>0 ); assert( pOp->p1<=p->nMem ); + assert( memIsValid(&aMem[pOp->p1]) ); REGISTER_TRACE(pOp->p1, &aMem[pOp->p1]); } if( (pOp->opflags & OPFLG_IN2)!=0 ){ assert( pOp->p2>0 ); assert( pOp->p2<=p->nMem ); + assert( memIsValid(&aMem[pOp->p2]) ); REGISTER_TRACE(pOp->p2, &aMem[pOp->p2]); } if( (pOp->opflags & OPFLG_IN3)!=0 ){ assert( pOp->p3>0 ); assert( pOp->p3<=p->nMem ); + assert( memIsValid(&aMem[pOp->p3]) ); REGISTER_TRACE(pOp->p3, &aMem[pOp->p3]); } if( (pOp->opflags & OPFLG_OUT2)!=0 ){ assert( pOp->p2>0 ); assert( pOp->p2<=p->nMem ); + memAboutToChange(p, &aMem[pOp->p2]); } if( (pOp->opflags & OPFLG_OUT3)!=0 ){ assert( pOp->p3>0 ); assert( pOp->p3<=p->nMem ); + memAboutToChange(p, &aMem[pOp->p3]); } #endif switch( pOp->opcode ){ @@ -59523,10 +60740,11 @@ ** and then jump to address P2. */ case OP_Gosub: { /* jump, in1 */ pIn1 = &aMem[pOp->p1]; assert( (pIn1->flags & MEM_Dyn)==0 ); + memAboutToChange(p, pIn1); pIn1->flags = MEM_Int; pIn1->u.i = pc; REGISTER_TRACE(pOp->p1, pIn1); pc = pOp->p2 - 1; break; @@ -59730,15 +60948,11 @@ /* Opcode: Blob P1 P2 * P4 ** ** P4 points to a blob of data P1 bytes long. Store this -** blob in register P2. This instruction is not coded directly -** by the compiler. Instead, the compiler layer specifies -** an OP_HexBlob opcode, with the hex string representation of -** the blob as P4. This opcode is transformed to an OP_Blob -** the first time it is executed. +** blob in register P2. */ case OP_Blob: { /* out2-prerelease */ assert( pOp->p1 <= SQLITE_MAX_LENGTH ); sqlite3VdbeMemSetStr(pOut, pOp->p4.z, pOp->p1, 0, 0); pOut->enc = encoding; @@ -59792,10 +61006,12 @@ pIn1 = &aMem[u.ac.p1]; pOut = &aMem[u.ac.p2]; while( u.ac.n-- ){ assert( pOut<=&aMem[p->nMem] ); assert( pIn1<=&aMem[p->nMem] ); + assert( memIsValid(pIn1) ); + memAboutToChange(p, pOut); u.ac.zMalloc = pOut->zMalloc; pOut->zMalloc = 0; sqlite3VdbeMemMove(pOut, pIn1); pIn1->zMalloc = u.ac.zMalloc; REGISTER_TRACE(u.ac.p2++, pOut); @@ -59837,10 +61053,13 @@ case OP_SCopy: { /* in1, out2 */ pIn1 = &aMem[pOp->p1]; pOut = &aMem[pOp->p2]; assert( pOut!=pIn1 ); sqlite3VdbeMemShallowCopy(pOut, pIn1, MEM_Ephem); +#ifdef SQLITE_DEBUG + if( pOut->pScopyFrom==0 ) pOut->pScopyFrom = pIn1; +#endif REGISTER_TRACE(pOp->p2, pOut); break; } /* Opcode: ResultRow P1 P2 * * * @@ -59897,10 +61116,14 @@ ** and have an assigned type. The results are de-ephemeralized as ** as side effect. */ u.ad.pMem = p->pResultSet = &aMem[pOp->p1]; for(u.ad.i=0; u.ad.i<pOp->p2; u.ad.i++){ + assert( memIsValid(&u.ad.pMem[u.ad.i]) ); + Deephemeralize(&u.ad.pMem[u.ad.i]); + assert( (u.ad.pMem[u.ad.i].flags & MEM_Ephem)==0 + || (u.ad.pMem[u.ad.i].flags & (MEM_Str|MEM_Blob))==0 ); sqlite3VdbeMemNulTerminate(&u.ad.pMem[u.ad.i]); sqlite3VdbeMemStoreType(&u.ad.pMem[u.ad.i]); REGISTER_TRACE(pOp->p1+u.ad.i, &u.ad.pMem[u.ad.i]); } if( db->mallocFailed ) goto no_mem; @@ -60128,16 +61351,21 @@ #endif /* local variables moved into u.ag */ u.ag.n = pOp->p5; u.ag.apVal = p->apArg; assert( u.ag.apVal || u.ag.n==0 ); + assert( pOp->p3>0 && pOp->p3<=p->nMem ); + pOut = &aMem[pOp->p3]; + memAboutToChange(p, pOut); assert( u.ag.n==0 || (pOp->p2>0 && pOp->p2+u.ag.n<=p->nMem+1) ); assert( pOp->p3<pOp->p2 || pOp->p3>=pOp->p2+u.ag.n ); u.ag.pArg = &aMem[pOp->p2]; for(u.ag.i=0; u.ag.i<u.ag.n; u.ag.i++, u.ag.pArg++){ + assert( memIsValid(u.ag.pArg) ); u.ag.apVal[u.ag.i] = u.ag.pArg; + Deephemeralize(u.ag.pArg); sqlite3VdbeMemStoreType(u.ag.pArg); REGISTER_TRACE(pOp->p2+u.ag.i, u.ag.pArg); } assert( pOp->p4type==P4_FUNCDEF || pOp->p4type==P4_VDBEFUNC ); @@ -60147,12 +61375,10 @@ }else{ u.ag.ctx.pVdbeFunc = (VdbeFunc*)pOp->p4.pVdbeFunc; u.ag.ctx.pFunc = u.ag.ctx.pVdbeFunc->pFunc; } - assert( pOp->p3>0 && pOp->p3<=p->nMem ); - pOut = &aMem[pOp->p3]; u.ag.ctx.s.flags = MEM_Null; u.ag.ctx.s.db = db; u.ag.ctx.s.xDel = 0; u.ag.ctx.s.zMalloc = 0; @@ -60168,11 +61394,11 @@ assert( pOp>aOp ); assert( pOp[-1].p4type==P4_COLLSEQ ); assert( pOp[-1].opcode==OP_CollSeq ); u.ag.ctx.pColl = pOp[-1].p4.pColl; } - (*u.ag.ctx.pFunc->xFunc)(&u.ag.ctx, u.ag.n, u.ag.apVal); + (*u.ag.ctx.pFunc->xFunc)(&u.ag.ctx, u.ag.n, u.ag.apVal); /* IMP: R-24505-23230 */ if( db->mallocFailed ){ /* Even though a malloc() has failed, the implementation of the ** user function may have called an sqlite3_result_XXX() function ** to return a value. The following call releases any resources ** associated with such a value. @@ -60220,11 +61446,11 @@ ** If either input is NULL, the result is NULL. */ /* Opcode: ShiftLeft P1 P2 P3 * * ** ** Shift the integer value in register P2 to the left by the -** number of bits specified by the integer in regiser P1. +** number of bits specified by the integer in register P1. ** Store the result in register P3. ** If either input is NULL, the result is NULL. */ /* Opcode: ShiftRight P1 P2 P3 * * ** @@ -60270,10 +61496,11 @@ ** ** To force any register to be an integer, just add 0. */ case OP_AddImm: { /* in1 */ pIn1 = &aMem[pOp->p1]; + memAboutToChange(p, pIn1); sqlite3VdbeMemIntegerify(pIn1); pIn1->u.i += pOp->p2; break; } @@ -60284,10 +61511,11 @@ ** without data loss, then jump immediately to P2, or if P2==0 ** raise an SQLITE_MISMATCH exception. */ case OP_MustBeInt: { /* jump, in1 */ pIn1 = &aMem[pOp->p1]; + memAboutToChange(p, pIn1); applyAffinity(pIn1, SQLITE_AFF_NUMERIC, encoding); if( (pIn1->flags & MEM_Int)==0 ){ if( pOp->p2==0 ){ rc = SQLITE_MISMATCH; goto abort_due_to_error; @@ -60329,10 +61557,11 @@ ** ** A NULL value is not changed by this routine. It remains NULL. */ case OP_ToText: { /* same as TK_TO_TEXT, in1 */ pIn1 = &aMem[pOp->p1]; + memAboutToChange(p, pIn1); if( pIn1->flags & MEM_Null ) break; assert( MEM_Str==(MEM_Blob>>3) ); pIn1->flags |= (pIn1->flags&MEM_Blob)>>3; applyAffinity(pIn1, SQLITE_AFF_TEXT, encoding); rc = ExpandBlob(pIn1); @@ -60375,20 +61604,18 @@ ** ** A NULL value is not changed by this routine. It remains NULL. */ case OP_ToNumeric: { /* same as TK_TO_NUMERIC, in1 */ pIn1 = &aMem[pOp->p1]; - if( (pIn1->flags & (MEM_Null|MEM_Int|MEM_Real))==0 ){ - sqlite3VdbeMemNumerify(pIn1); - } + sqlite3VdbeMemNumerify(pIn1); break; } #endif /* SQLITE_OMIT_CAST */ /* Opcode: ToInt P1 * * * * ** -** Force the value in register P1 be an integer. If +** Force the value in register P1 to be an integer. If ** The value is currently a real number, drop its fractional part. ** If the value is text or blob, try to convert it to an integer using the ** equivalent of atoi() and store 0 if no such conversion is possible. ** ** A NULL value is not changed by this routine. It remains NULL. @@ -60411,10 +61638,11 @@ ** ** A NULL value is not changed by this routine. It remains NULL. */ case OP_ToReal: { /* same as TK_TO_REAL, in1 */ pIn1 = &aMem[pOp->p1]; + memAboutToChange(p, pIn1); if( (pIn1->flags & MEM_Null)==0 ){ sqlite3VdbeMemRealify(pIn1); } break; } @@ -60425,11 +61653,11 @@ ** Compare the values in register P1 and P3. If reg(P3)<reg(P1) then ** jump to address P2. ** ** If the SQLITE_JUMPIFNULL bit of P5 is set and either reg(P1) or ** reg(P3) is NULL then take the jump. If the SQLITE_JUMPIFNULL -** bit is clear then fall thru if either operand is NULL. +** bit is clear then fall through if either operand is NULL. ** ** The SQLITE_AFF_MASK portion of P5 must be an affinity character - ** SQLITE_AFF_TEXT, SQLITE_AFF_INTEGER, and so forth. An attempt is made ** to coerce both inputs according to this affinity before the ** comparison is made. If the SQLITE_AFF_MASK is 0x00, then numeric @@ -60555,10 +61783,11 @@ default: u.ai.res = u.ai.res>=0; break; } if( pOp->p5 & SQLITE_STOREP2 ){ pOut = &aMem[pOp->p2]; + memAboutToChange(p, pOut); MemSetTypeFlag(pOut, MEM_Int); pOut->u.i = u.ai.res; REGISTER_TRACE(pOp->p2, pOut); }else if( u.ai.res ){ pc = pOp->p2-1; @@ -60586,12 +61815,12 @@ break; } /* Opcode: Compare P1 P2 P3 P4 * ** -** Compare to vectors of registers in reg(P1)..reg(P1+P3-1) (all this -** one "A") and in reg(P2)..reg(P2+P3-1) ("B"). Save the result of +** Compare two vectors of registers in reg(P1)..reg(P1+P3-1) (call this +** vector "A") and in reg(P2)..reg(P2+P3-1) ("B"). Save the result of ** the comparison for use by the next OP_Jump instruct. ** ** P4 is a KeyInfo structure that defines collating sequences and sort ** orders for the comparison. The permutation applies to registers ** only. The KeyInfo elements are used sequentially. @@ -60629,10 +61858,12 @@ assert( u.aj.p2>0 && u.aj.p2+u.aj.n<=p->nMem+1 ); } #endif /* SQLITE_DEBUG */ for(u.aj.i=0; u.aj.i<u.aj.n; u.aj.i++){ u.aj.idx = aPermute ? aPermute[u.aj.i] : u.aj.i; + assert( memIsValid(&aMem[u.aj.p1+u.aj.idx]) ); + assert( memIsValid(&aMem[u.aj.p2+u.aj.idx]) ); REGISTER_TRACE(u.aj.p1+u.aj.idx, &aMem[u.aj.p1+u.aj.idx]); REGISTER_TRACE(u.aj.p2+u.aj.idx, &aMem[u.aj.p2+u.aj.idx]); assert( u.aj.i<u.aj.pKeyInfo->nField ); u.aj.pColl = u.aj.pKeyInfo->aColl[u.aj.i]; u.aj.bRev = u.aj.pKeyInfo->aSortOrder[u.aj.i]; @@ -60860,10 +62091,11 @@ u.am.pC = 0; memset(&u.am.sMem, 0, sizeof(u.am.sMem)); assert( u.am.p1<p->nCursor ); assert( pOp->p3>0 && pOp->p3<=p->nMem ); u.am.pDest = &aMem[pOp->p3]; + memAboutToChange(p, u.am.pDest); MemSetTypeFlag(u.am.pDest, MEM_Null); u.am.zRec = 0; /* This block sets the variable u.am.payloadSize to be the total number of ** bytes in the record. @@ -60907,10 +62139,11 @@ assert( rc==SQLITE_OK ); /* DataSize() cannot fail */ } }else if( u.am.pC->pseudoTableReg>0 ){ u.am.pReg = &aMem[u.am.pC->pseudoTableReg]; assert( u.am.pReg->flags & MEM_Blob ); + assert( memIsValid(u.am.pReg) ); u.am.payloadSize = u.am.pReg->n; u.am.zRec = u.am.pReg->z; u.am.pC->cacheStatus = (pOp->p5&OPFLAG_CLEARCACHE) ? CACHE_STALE : p->cacheCtr; assert( u.am.payloadSize==0 || u.am.zRec!=0 ); }else{ @@ -61131,25 +62364,23 @@ assert( u.an.zAffinity!=0 ); assert( u.an.zAffinity[pOp->p2]==0 ); pIn1 = &aMem[pOp->p1]; while( (u.an.cAff = *(u.an.zAffinity++))!=0 ){ assert( pIn1 <= &p->aMem[p->nMem] ); + assert( memIsValid(pIn1) ); ExpandBlob(pIn1); applyAffinity(pIn1, u.an.cAff, encoding); pIn1++; } break; } /* Opcode: MakeRecord P1 P2 P3 P4 * ** -** Convert P2 registers beginning with P1 into a single entry -** suitable for use as a data record in a database table or as a key -** in an index. The details of the format are irrelevant as long as -** the OP_Column opcode can decode the record later. -** Refer to source code comments for the details of the record -** format. +** Convert P2 registers beginning with P1 into the [record format] +** use as a data record in a database table or as a key +** in an index. The OP_Column opcode can decode the record later. ** ** P4 may be a string that is P2 characters long. The nth character of the ** string indicates the column affinity that should be used for the nth ** field of the index key. ** @@ -61201,15 +62432,21 @@ assert( u.ao.nField>0 && pOp->p2>0 && pOp->p2+u.ao.nField<=p->nMem+1 ); u.ao.pData0 = &aMem[u.ao.nField]; u.ao.nField = pOp->p2; u.ao.pLast = &u.ao.pData0[u.ao.nField-1]; u.ao.file_format = p->minWriteFileFormat; + + /* Identify the output register */ + assert( pOp->p3<pOp->p1 || pOp->p3>=pOp->p1+pOp->p2 ); + pOut = &aMem[pOp->p3]; + memAboutToChange(p, pOut); /* Loop through the elements that will make up the record to figure ** out how much space is required for the new record. */ for(u.ao.pRec=u.ao.pData0; u.ao.pRec<=u.ao.pLast; u.ao.pRec++){ + assert( memIsValid(u.ao.pRec) ); if( u.ao.zAffinity ){ applyAffinity(u.ao.pRec, u.ao.zAffinity[u.ao.pRec-u.ao.pData0], encoding); } if( u.ao.pRec->flags&MEM_Zero && u.ao.pRec->n>0 ){ sqlite3VdbeMemExpandBlob(u.ao.pRec); @@ -61240,12 +62477,10 @@ /* Make sure the output register has a buffer large enough to store ** the new record. The output register (pOp->p3) is not allowed to ** be one of the input registers (because the following call to ** sqlite3VdbeMemGrow() could clobber the value before it is used). */ - assert( pOp->p3<pOp->p1 || pOp->p3>=pOp->p1+pOp->p2 ); - pOut = &aMem[pOp->p3]; if( sqlite3VdbeMemGrow(pOut, (int)u.ao.nByte, 0) ){ goto no_mem; } u.ao.zNewRecord = (u8 *)pOut->z; @@ -61414,10 +62649,11 @@ } } if( u.aq.p1==SAVEPOINT_ROLLBACK && (db->flags&SQLITE_InternChanges)!=0 ){ sqlite3ExpirePreparedStatements(db); sqlite3ResetInternalSchema(db, 0); + db->flags = (db->flags | SQLITE_InternChanges); } } /* Regardless of whether this is a RELEASE or ROLLBACK, destroy all ** savepoints nested inside of the savepoint being operated on. */ @@ -61804,10 +63040,12 @@ } if( pOp->p5 ){ assert( u.aw.p2>0 ); assert( u.aw.p2<=p->nMem ); pIn2 = &aMem[u.aw.p2]; + assert( memIsValid(pIn2) ); + assert( (pIn2->flags & MEM_Int)!=0 ); sqlite3VdbeMemIntegerify(pIn2); u.aw.p2 = (int)pIn2->u.i; /* The u.aw.p2 value always comes from a prior OP_CreateTable opcode and ** that opcode will always set the u.aw.p2 value to 2 or more or else fail. ** If there were a failure, the prepared statement would have halted @@ -61826,10 +63064,11 @@ } assert( pOp->p1>=0 ); u.aw.pCur = allocateCursor(p, pOp->p1, u.aw.nField, u.aw.iDb, 1); if( u.aw.pCur==0 ) goto no_mem; u.aw.pCur->nullRow = 1; + u.aw.pCur->isOrdered = 1; rc = sqlite3BtreeCursor(u.aw.pX, u.aw.p2, u.aw.wrFlag, u.aw.pKeyInfo, u.aw.pCur->pCursor); u.aw.pCur->pKeyInfo = u.aw.pKeyInfo; /* Since it performs no memory allocation or IO, the only values that ** sqlite3BtreeCursor() may return are SQLITE_EMPTY and SQLITE_OK. @@ -61878,11 +63117,11 @@ case OP_OpenAutoindex: case OP_OpenEphemeral: { #if 0 /* local variables moved into u.ax */ VdbeCursor *pCx; #endif /* local variables moved into u.ax */ - static const int openFlags = + static const int vfsFlags = SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE | SQLITE_OPEN_EXCLUSIVE | SQLITE_OPEN_DELETEONCLOSE | SQLITE_OPEN_TRANSIENT_DB; @@ -61889,25 +63128,25 @@ assert( pOp->p1>=0 ); u.ax.pCx = allocateCursor(p, pOp->p1, pOp->p2, -1, 1); if( u.ax.pCx==0 ) goto no_mem; u.ax.pCx->nullRow = 1; - rc = sqlite3BtreeFactory(db, 0, 1, SQLITE_DEFAULT_TEMP_CACHE_SIZE, openFlags, - &u.ax.pCx->pBt); + rc = sqlite3BtreeOpen(0, db, &u.ax.pCx->pBt, + BTREE_OMIT_JOURNAL | BTREE_SINGLE | pOp->p5, vfsFlags); if( rc==SQLITE_OK ){ rc = sqlite3BtreeBeginTrans(u.ax.pCx->pBt, 1); } if( rc==SQLITE_OK ){ /* If a transient index is required, create it by calling - ** sqlite3BtreeCreateTable() with the BTREE_ZERODATA flag before + ** sqlite3BtreeCreateTable() with the BTREE_BLOBKEY flag before ** opening it. If a transient table is required, just use the - ** automatically created table with root-page 1 (an INTKEY table). + ** automatically created table with root-page 1 (an BLOB_INTKEY table). */ if( pOp->p4.pKeyInfo ){ int pgno; assert( pOp->p4type==P4_KEYINFO ); - rc = sqlite3BtreeCreateTable(u.ax.pCx->pBt, &pgno, BTREE_ZERODATA); + rc = sqlite3BtreeCreateTable(u.ax.pCx->pBt, &pgno, BTREE_BLOBKEY); if( rc==SQLITE_OK ){ assert( pgno==MASTER_ROOT+1 ); rc = sqlite3BtreeCursor(u.ax.pCx->pBt, pgno, 1, (KeyInfo*)pOp->p4.z, u.ax.pCx->pCursor); u.ax.pCx->pKeyInfo = pOp->p4.pKeyInfo; @@ -61917,10 +63156,11 @@ }else{ rc = sqlite3BtreeCursor(u.ax.pCx->pBt, MASTER_ROOT, 1, 0, u.ax.pCx->pCursor); u.ax.pCx->isTable = 1; } } + u.ax.pCx->isOrdered = (pOp->p5!=BTREE_UNORDERED); u.ax.pCx->isIndex = !u.ax.pCx->isTable; break; } /* Opcode: OpenPseudo P1 P2 P3 * * @@ -62036,10 +63276,11 @@ assert( u.az.pC!=0 ); assert( u.az.pC->pseudoTableReg==0 ); assert( OP_SeekLe == OP_SeekLt+1 ); assert( OP_SeekGe == OP_SeekLt+2 ); assert( OP_SeekGt == OP_SeekLt+3 ); + assert( u.az.pC->isOrdered ); if( u.az.pC->pCursor!=0 ){ u.az.oc = pOp->opcode; u.az.pC->nullRow = 0; if( u.az.pC->isTable ){ /* The input value in P3 might be of any type: integer, real, string, @@ -62118,10 +63359,13 @@ assert( u.az.oc!=OP_SeekLe || u.az.r.flags==UNPACKED_INCRKEY ); assert( u.az.oc!=OP_SeekGe || u.az.r.flags==0 ); assert( u.az.oc!=OP_SeekLt || u.az.r.flags==0 ); u.az.r.aMem = &aMem[pOp->p3]; +#ifdef SQLITE_DEBUG + { int i; for(i=0; i<u.az.r.nField; i++) assert( memIsValid(&u.az.r.aMem[i]) ); } +#endif ExpandBlob(u.az.r.aMem); rc = sqlite3BtreeMovetoUnpacked(u.az.pC->pCursor, &u.az.r, 0, 0, &u.az.res); if( rc!=SQLITE_OK ){ goto abort_due_to_error; } @@ -62246,15 +63490,18 @@ assert( u.bb.pC->isTable==0 ); if( pOp->p4.i>0 ){ u.bb.r.pKeyInfo = u.bb.pC->pKeyInfo; u.bb.r.nField = (u16)pOp->p4.i; u.bb.r.aMem = pIn3; +#ifdef SQLITE_DEBUG + { int i; for(i=0; i<u.bb.r.nField; i++) assert( memIsValid(&u.bb.r.aMem[i]) ); } +#endif u.bb.r.flags = UNPACKED_PREFIX_MATCH; u.bb.pIdxKey = &u.bb.r; }else{ assert( pIn3->flags & MEM_Blob ); - ExpandBlob(pIn3); + assert( (pIn3->flags & MEM_Zero)==0 ); /* zeroblobs already expanded */ u.bb.pIdxKey = sqlite3VdbeRecordUnpack(u.bb.pC->pKeyInfo, pIn3->n, pIn3->z, u.bb.aTempRec, sizeof(u.bb.aTempRec)); if( u.bb.pIdxKey==0 ){ goto no_mem; } @@ -62345,10 +63592,13 @@ /* Populate the index search key. */ u.bc.r.pKeyInfo = u.bc.pCx->pKeyInfo; u.bc.r.nField = u.bc.nField + 1; u.bc.r.flags = UNPACKED_PREFIX_SEARCH; u.bc.r.aMem = u.bc.aMx; +#ifdef SQLITE_DEBUG + { int i; for(i=0; i<u.bc.r.nField; i++) assert( memIsValid(&u.bc.r.aMem[i]) ); } +#endif /* Extract the value of u.bc.R from register P3. */ sqlite3VdbeMemIntegerify(pIn3); u.bc.R = pIn3->u.i; @@ -62367,11 +63617,11 @@ /* Opcode: NotExists P1 P2 P3 * * ** ** Use the content of register P3 as a integer key. If a record ** with that key does not exist in table of P1, then jump to P2. -** If the record does exist, then fall thru. The cursor is left +** If the record does exist, then fall through. The cursor is left ** pointing to the record if it exists. ** ** The difference between this operation and NotFound is that this ** operation assumes the key is an integer and that P1 is a table whereas ** NotFound assumes key is a blob constructed from MakeRecord and @@ -62525,11 +63775,13 @@ u.be.pMem = &u.be.pFrame->aMem[pOp->p3]; }else{ /* Assert that P3 is a valid memory cell. */ assert( pOp->p3<=p->nMem ); u.be.pMem = &aMem[pOp->p3]; + memAboutToChange(p, u.be.pMem); } + assert( memIsValid(u.be.pMem) ); REGISTER_TRACE(pOp->p3, u.be.pMem); sqlite3VdbeMemIntegerify(u.be.pMem); assert( (u.be.pMem->flags & MEM_Int)!=0 ); /* mem(P3) holds an integer */ if( u.be.pMem->u.i==MAX_ROWID || u.be.pC->useRandomRowid ){ @@ -62544,33 +63796,40 @@ #endif sqlite3BtreeSetCachedRowid(u.be.pC->pCursor, u.be.v<MAX_ROWID ? u.be.v+1 : 0); } if( u.be.pC->useRandomRowid ){ - /* IMPLEMENTATION-OF: R-48598-02938 If the largest ROWID is equal to the + /* IMPLEMENTATION-OF: R-07677-41881 If the largest ROWID is equal to the ** largest possible integer (9223372036854775807) then the database - ** engine starts picking candidate ROWIDs at random until it finds one - ** that is not previously used. - */ + ** engine starts picking positive candidate ROWIDs at random until + ** it finds one that is not previously used. */ assert( pOp->p3==0 ); /* We cannot be in random rowid mode if this is ** an AUTOINCREMENT table. */ + /* on the first attempt, simply do one more than previous */ u.be.v = db->lastRowid; + u.be.v &= (MAX_ROWID>>1); /* ensure doesn't go negative */ + u.be.v++; /* ensure non-zero */ u.be.cnt = 0; - do{ - if( u.be.cnt==0 && (u.be.v&0xffffff)==u.be.v ){ - u.be.v++; + while( ((rc = sqlite3BtreeMovetoUnpacked(u.be.pC->pCursor, 0, (u64)u.be.v, + 0, &u.be.res))==SQLITE_OK) + && (u.be.res==0) + && (++u.be.cnt<100)){ + /* collision - try another random rowid */ + sqlite3_randomness(sizeof(u.be.v), &u.be.v); + if( u.be.cnt<5 ){ + /* try "small" random rowids for the initial attempts */ + u.be.v &= 0xffffff; }else{ - sqlite3_randomness(sizeof(u.be.v), &u.be.v); - if( u.be.cnt<5 ) u.be.v &= 0xffffff; + u.be.v &= (MAX_ROWID>>1); /* ensure doesn't go negative */ } - rc = sqlite3BtreeMovetoUnpacked(u.be.pC->pCursor, 0, (u64)u.be.v, 0, &u.be.res); - u.be.cnt++; - }while( u.be.cnt<100 && rc==SQLITE_OK && u.be.res==0 ); + u.be.v++; /* ensure non-zero */ + } if( rc==SQLITE_OK && u.be.res==0 ){ rc = SQLITE_FULL; /* IMP: R-38219-53002 */ goto abort_due_to_error; } + assert( u.be.v>0 ); /* EV: R-40812-03570 */ } u.be.pC->rowidIsValid = 0; u.be.pC->deferredMoveto = 0; u.be.pC->cacheStatus = CACHE_STALE; } @@ -62636,10 +63895,11 @@ int op; /* Opcode for update hook: SQLITE_UPDATE or SQLITE_INSERT */ #endif /* local variables moved into u.bf */ u.bf.pData = &aMem[pOp->p2]; assert( pOp->p1>=0 && pOp->p1<p->nCursor ); + assert( memIsValid(u.bf.pData) ); u.bf.pC = p->apCsr[pOp->p1]; assert( u.bf.pC!=0 ); assert( u.bf.pC->pCursor!=0 ); assert( u.bf.pC->pseudoTableReg==0 ); assert( u.bf.pC->isTable ); @@ -62646,10 +63906,11 @@ REGISTER_TRACE(pOp->p2, u.bf.pData); if( pOp->opcode==OP_Insert ){ u.bf.pKey = &aMem[pOp->p3]; assert( u.bf.pKey->flags & MEM_Int ); + assert( memIsValid(u.bf.pKey) ); REGISTER_TRACE(pOp->p3, u.bf.pKey); u.bf.iKey = u.bf.pKey->u.i; }else{ assert( pOp->opcode==OP_InsertInt ); u.bf.iKey = pOp->p3; @@ -62797,10 +64058,11 @@ u32 n; i64 n64; #endif /* local variables moved into u.bh */ pOut = &aMem[pOp->p2]; + memAboutToChange(p, pOut); /* Note that RowKey and RowData are really exactly the same instruction */ assert( pOp->p1>=0 && pOp->p1<p->nCursor ); u.bh.pC = p->apCsr[pOp->p1]; assert( u.bh.pC->isTable || pOp->opcode==OP_RowKey ); @@ -63139,10 +64401,13 @@ if( ALWAYS(u.bo.pCrsr!=0) ){ u.bo.r.pKeyInfo = u.bo.pC->pKeyInfo; u.bo.r.nField = (u16)pOp->p3; u.bo.r.flags = 0; u.bo.r.aMem = &aMem[pOp->p2]; +#ifdef SQLITE_DEBUG + { int i; for(i=0; i<u.bo.r.nField; i++) assert( memIsValid(&u.bo.r.aMem[i]) ); } +#endif rc = sqlite3BtreeMovetoUnpacked(u.bo.pCrsr, &u.bo.r, 0, 0, &u.bo.res); if( rc==SQLITE_OK && u.bo.res==0 ){ rc = sqlite3BtreeDelete(u.bo.pCrsr); } assert( u.bo.pC->deferredMoveto==0 ); @@ -63200,11 +64465,11 @@ ** If P5 is non-zero then the key value is increased by an epsilon ** prior to the comparison. This make the opcode work like IdxGT except ** that if the key from register P3 is a prefix of the key in the cursor, ** the result is false whereas it would be true with IdxGT. */ -/* Opcode: IdxLT P1 P2 P3 * P5 +/* Opcode: IdxLT P1 P2 P3 P4 P5 ** ** The P4 register values beginning with P3 form an unpacked index ** key that omits the ROWID. Compare this key value against the index ** that P1 is currently pointing to, ignoring the ROWID on the P1 index. ** @@ -63223,10 +64488,11 @@ #endif /* local variables moved into u.bq */ assert( pOp->p1>=0 && pOp->p1<p->nCursor ); u.bq.pC = p->apCsr[pOp->p1]; assert( u.bq.pC!=0 ); + assert( u.bq.pC->isOrdered ); if( ALWAYS(u.bq.pC->pCursor!=0) ){ assert( u.bq.pC->deferredMoveto==0 ); assert( pOp->p5==0 || pOp->p5==1 ); assert( pOp->p4type==P4_INT32 ); u.bq.r.pKeyInfo = u.bq.pC->pKeyInfo; @@ -63235,10 +64501,13 @@ u.bq.r.flags = UNPACKED_INCRKEY | UNPACKED_IGNORE_ROWID; }else{ u.bq.r.flags = UNPACKED_IGNORE_ROWID; } u.bq.r.aMem = &aMem[pOp->p3]; +#ifdef SQLITE_DEBUG + { int i; for(i=0; i<u.bq.r.nField; i++) assert( memIsValid(&u.bq.r.aMem[i]) ); } +#endif rc = sqlite3VdbeIdxKeyCompare(u.bq.pC, &u.bq.r, &u.bq.res); if( pOp->opcode==OP_IdxLT ){ u.bq.res = -u.bq.res; }else{ assert( pOp->opcode==OP_IdxGE ); @@ -63338,10 +64607,12 @@ db->aDb[pOp->p2].pBt, pOp->p1, (pOp->p3 ? &u.bs.nChange : 0) ); if( pOp->p3 ){ p->nChange += u.bs.nChange; if( pOp->p3>0 ){ + assert( memIsValid(&aMem[pOp->p3]) ); + memAboutToChange(p, &aMem[pOp->p3]); aMem[pOp->p3].u.i += u.bs.nChange; } } break; } @@ -63381,13 +64652,13 @@ assert( (p->btreeMask & (1<<pOp->p1))!=0 ); u.bt.pDb = &db->aDb[pOp->p1]; assert( u.bt.pDb->pBt!=0 ); if( pOp->opcode==OP_CreateTable ){ /* u.bt.flags = BTREE_INTKEY; */ - u.bt.flags = BTREE_LEAFDATA|BTREE_INTKEY; + u.bt.flags = BTREE_INTKEY; }else{ - u.bt.flags = BTREE_ZERODATA; + u.bt.flags = BTREE_BLOBKEY; } rc = sqlite3BtreeCreateTable(u.bt.pDb->pBt, &u.bt.pgno, u.bt.flags); pOut->u.i = u.bt.pgno; break; } @@ -63712,10 +64983,11 @@ void *t; /* Token identifying trigger */ #endif /* local variables moved into u.by */ u.by.pProgram = pOp->p4.pProgram; u.by.pRt = &aMem[pOp->p3]; + assert( memIsValid(u.by.pRt) ); assert( u.by.pProgram->nOp>0 ); /* If the p5 flag is clear, then recursive invocation of triggers is ** disabled for backwards compatibility (p5 is set if this sub-program ** is really a trigger, not a foreign key action, and the flag set @@ -63885,10 +65157,11 @@ for(u.ca.pFrame=p->pFrame; u.ca.pFrame->pParent; u.ca.pFrame=u.ca.pFrame->pParent); u.ca.pIn1 = &u.ca.pFrame->aMem[pOp->p1]; }else{ u.ca.pIn1 = &aMem[pOp->p1]; } + assert( memIsValid(u.ca.pIn1) ); sqlite3VdbeMemIntegerify(u.ca.pIn1); pIn2 = &aMem[pOp->p2]; sqlite3VdbeMemIntegerify(pIn2); if( u.ca.pIn1->u.i<pIn2->u.i){ u.ca.pIn1->u.i = pIn2->u.i; @@ -63971,11 +65244,13 @@ assert( u.cb.n>=0 ); u.cb.pRec = &aMem[pOp->p2]; u.cb.apVal = p->apArg; assert( u.cb.apVal || u.cb.n==0 ); for(u.cb.i=0; u.cb.i<u.cb.n; u.cb.i++, u.cb.pRec++){ + assert( memIsValid(u.cb.pRec) ); u.cb.apVal[u.cb.i] = u.cb.pRec; + memAboutToChange(p, u.cb.pRec); sqlite3VdbeMemStoreType(u.cb.pRec); } u.cb.ctx.pFunc = pOp->p4.pFunc; assert( pOp->p3>0 && pOp->p3<=p->nMem ); u.cb.ctx.pMem = u.cb.pMem = &aMem[pOp->p3]; @@ -63991,11 +65266,11 @@ assert( pOp>p->aOp ); assert( pOp[-1].p4type==P4_COLLSEQ ); assert( pOp[-1].opcode==OP_CollSeq ); u.cb.ctx.pColl = pOp[-1].p4.pColl; } - (u.cb.ctx.pFunc->xStep)(&u.cb.ctx, u.cb.n, u.cb.apVal); + (u.cb.ctx.pFunc->xStep)(&u.cb.ctx, u.cb.n, u.cb.apVal); /* IMP: R-24505-23230 */ if( u.cb.ctx.isError ){ sqlite3SetString(&p->zErrMsg, db, "%s", sqlite3_value_text(&u.cb.ctx.s)); rc = u.cb.ctx.isError; } sqlite3VdbeMemRelease(&u.cb.ctx.s); @@ -64378,10 +65653,11 @@ #endif /* local variables moved into u.ch */ u.ch.pQuery = &aMem[pOp->p3]; u.ch.pArgc = &u.ch.pQuery[1]; u.ch.pCur = p->apCsr[pOp->p1]; + assert( memIsValid(u.ch.pQuery) ); REGISTER_TRACE(pOp->p3, u.ch.pQuery); assert( u.ch.pCur->pVtabCursor ); u.ch.pVtabCursor = u.ch.pCur->pVtabCursor; u.ch.pVtab = u.ch.pVtabCursor->pVtab; u.ch.pModule = u.ch.pVtab->pModule; @@ -64435,10 +65711,11 @@ VdbeCursor *pCur = p->apCsr[pOp->p1]; assert( pCur->pVtabCursor ); assert( pOp->p3>0 && pOp->p3<=p->nMem ); u.ci.pDest = &aMem[pOp->p3]; + memAboutToChange(p, u.ci.pDest); if( pCur->nullRow ){ sqlite3VdbeMemSetNull(u.ci.pDest); break; } u.ci.pVtab = pCur->pVtabCursor->pVtab; @@ -64537,14 +65814,16 @@ #endif /* local variables moved into u.ck */ u.ck.pVtab = pOp->p4.pVtab->pVtab; u.ck.pName = &aMem[pOp->p1]; assert( u.ck.pVtab->pModule->xRename ); + assert( memIsValid(u.ck.pName) ); REGISTER_TRACE(pOp->p1, u.ck.pName); assert( u.ck.pName->flags & MEM_Str ); rc = u.ck.pVtab->pModule->xRename(u.ck.pVtab, u.ck.pName->z); importVtabErrMsg(p, u.ck.pVtab); + p->expired = 0; break; } #endif @@ -64589,10 +65868,12 @@ assert( pOp->p4type==P4_VTAB ); if( ALWAYS(u.cl.pModule->xUpdate) ){ u.cl.apArg = p->apArg; u.cl.pX = &aMem[pOp->p3]; for(u.cl.i=0; u.cl.i<u.cl.nArg; u.cl.i++){ + assert( memIsValid(u.cl.pX) ); + memAboutToChange(p, u.cl.pX); sqlite3VdbeMemStoreType(u.cl.pX); u.cl.apArg[u.cl.i] = u.cl.pX; u.cl.pX++; } rc = u.cl.pModule->xUpdate(u.cl.pVtab, u.cl.nArg, u.cl.apArg, &u.cl.rowid); @@ -65643,12 +66924,11 @@ SQLITE_PRIVATE int sqlite3IsMemJournal(sqlite3_file *pJfd){ return pJfd->pMethods==&MemJournalMethods; } /* -** Return the number of bytes required to store a MemJournal that uses vfs -** pVfs to create the underlying on-disk files. +** Return the number of bytes required to store a MemJournal file descriptor. */ SQLITE_PRIVATE int sqlite3MemJournalSize(void){ return sizeof(MemJournal); } @@ -67547,22 +68827,23 @@ assert( z[0]=='?' ); pExpr->iColumn = (ynVar)(++pParse->nVar); }else if( z[0]=='?' ){ /* Wildcard of the form "?nnn". Convert "nnn" to an integer and ** use it as the variable number */ - int i = atoi((char*)&z[1]); + i64 i; + int bOk = 0==sqlite3Atoi64(&z[1], &i, sqlite3Strlen30(&z[1]), SQLITE_UTF8); pExpr->iColumn = (ynVar)i; testcase( i==0 ); testcase( i==1 ); testcase( i==db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER]-1 ); testcase( i==db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ); - if( i<1 || i>db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ){ + if( bOk==0 || i<1 || i>db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER] ){ sqlite3ErrorMsg(pParse, "variable number must be between ?1 and ?%d", db->aLimit[SQLITE_LIMIT_VARIABLE_NUMBER]); } if( i>pParse->nVar ){ - pParse->nVar = i; + pParse->nVar = (int)i; } }else{ /* Wildcards like ":aaa", "$aaa" or "@aaa". Reuse the same variable ** number as the prior appearance of the same name, or if the name ** has never appeared before, reuse the same variable number @@ -68527,12 +69808,12 @@ return eType; } #endif /* -** Generate code for scalar subqueries used as an expression -** and IN operators. Examples: +** Generate code for scalar subqueries used as a subquery expression, EXISTS, +** or IN operators. Examples: ** ** (SELECT a FROM b) -- subquery ** EXISTS (SELECT a FROM b) -- EXISTS subquery ** x IN (4,5,11) -- IN operator with list on right-hand side ** x IN (SELECT a FROM b) -- IN operator with subquery on the right @@ -68591,14 +69872,14 @@ assert( testAddr>0 || pParse->db->mallocFailed ); } switch( pExpr->op ){ case TK_IN: { - char affinity; - KeyInfo keyInfo; - int addr; /* Address of OP_OpenEphemeral instruction */ - Expr *pLeft = pExpr->pLeft; + char affinity; /* Affinity of the LHS of the IN */ + KeyInfo keyInfo; /* Keyinfo for the generated table */ + int addr; /* Address of OP_OpenEphemeral instruction */ + Expr *pLeft = pExpr->pLeft; /* the LHS of the IN operator */ if( rMayHaveNull ){ sqlite3VdbeAddOp2(v, OP_Null, 0, rMayHaveNull); } @@ -68617,10 +69898,11 @@ ** 'x' nor the SELECT... statement are columns, then numeric affinity ** is used. */ pExpr->iTable = pParse->nTab++; addr = sqlite3VdbeAddOp2(v, OP_OpenEphemeral, pExpr->iTable, !isRowid); + if( rMayHaveNull==0 ) sqlite3VdbeChangeP5(v, BTREE_UNORDERED); memset(&keyInfo, 0, sizeof(keyInfo)); keyInfo.nField = 1; if( ExprHasProperty(pExpr, EP_xIsSelect) ){ /* Case 1: expr IN (SELECT ...) @@ -68909,11 +70191,11 @@ */ static void codeReal(Vdbe *v, const char *z, int negateFlag, int iMem){ if( ALWAYS(z!=0) ){ double value; char *zV; - sqlite3AtoF(z, &value); + sqlite3AtoF(z, &value, sqlite3Strlen30(z), SQLITE_UTF8); assert( !sqlite3IsNaN(value) ); /* The new AtoF never returns NaN */ if( negateFlag ) value = -value; zV = dup8bytes(v, (char*)&value); sqlite3VdbeAddOp4(v, OP_Real, 0, iMem, 0, zV, P4_REAL); } @@ -68923,28 +70205,27 @@ /* ** Generate an instruction that will put the integer describe by ** text z[0..n-1] into register iMem. ** -** The z[] string will probably not be zero-terminated. But the -** z[n] character is guaranteed to be something that does not look -** like the continuation of the number. +** Expr.u.zToken is always UTF8 and zero-terminated. */ static void codeInteger(Parse *pParse, Expr *pExpr, int negFlag, int iMem){ Vdbe *v = pParse->pVdbe; if( pExpr->flags & EP_IntValue ){ int i = pExpr->u.iValue; if( negFlag ) i = -i; sqlite3VdbeAddOp2(v, OP_Integer, i, iMem); }else{ + int c; + i64 value; const char *z = pExpr->u.zToken; assert( z!=0 ); - if( sqlite3FitsIn64Bits(z, negFlag) ){ - i64 value; + c = sqlite3Atoi64(z, &value, sqlite3Strlen30(z), SQLITE_UTF8); + if( c==0 || (c==2 && negFlag) ){ char *zV; - sqlite3Atoi64(z, &value); - if( negFlag ) value = -value; + if( negFlag ){ value = -value; } zV = dup8bytes(v, (char*)&value); sqlite3VdbeAddOp4(v, OP_Int64, 0, iMem, 0, zV, P4_INT64); }else{ #ifdef SQLITE_OMIT_FLOATING_POINT sqlite3ErrorMsg(pParse, "oversized integer: %s%s", negFlag ? "-" : "", z); @@ -69225,77 +70506,10 @@ } return 0; } #endif /* SQLITE_DEBUG || SQLITE_COVERAGE_TEST */ -/* -** If the last instruction coded is an ephemeral copy of any of -** the registers in the nReg registers beginning with iReg, then -** convert the last instruction from OP_SCopy to OP_Copy. -*/ -SQLITE_PRIVATE void sqlite3ExprHardCopy(Parse *pParse, int iReg, int nReg){ - VdbeOp *pOp; - Vdbe *v; - - assert( pParse->db->mallocFailed==0 ); - v = pParse->pVdbe; - assert( v!=0 ); - pOp = sqlite3VdbeGetOp(v, -1); - assert( pOp!=0 ); - if( pOp->opcode==OP_SCopy && pOp->p1>=iReg && pOp->p1<iReg+nReg ){ - pOp->opcode = OP_Copy; - } -} - -/* -** Generate code to store the value of the iAlias-th alias in register -** target. The first time this is called, pExpr is evaluated to compute -** the value of the alias. The value is stored in an auxiliary register -** and the number of that register is returned. On subsequent calls, -** the register number is returned without generating any code. -** -** Note that in order for this to work, code must be generated in the -** same order that it is executed. -** -** Aliases are numbered starting with 1. So iAlias is in the range -** of 1 to pParse->nAlias inclusive. -** -** pParse->aAlias[iAlias-1] records the register number where the value -** of the iAlias-th alias is stored. If zero, that means that the -** alias has not yet been computed. -*/ -static int codeAlias(Parse *pParse, int iAlias, Expr *pExpr, int target){ -#if 0 - sqlite3 *db = pParse->db; - int iReg; - if( pParse->nAliasAlloc<pParse->nAlias ){ - pParse->aAlias = sqlite3DbReallocOrFree(db, pParse->aAlias, - sizeof(pParse->aAlias[0])*pParse->nAlias ); - testcase( db->mallocFailed && pParse->nAliasAlloc>0 ); - if( db->mallocFailed ) return 0; - memset(&pParse->aAlias[pParse->nAliasAlloc], 0, - (pParse->nAlias-pParse->nAliasAlloc)*sizeof(pParse->aAlias[0])); - pParse->nAliasAlloc = pParse->nAlias; - } - assert( iAlias>0 && iAlias<=pParse->nAlias ); - iReg = pParse->aAlias[iAlias-1]; - if( iReg==0 ){ - if( pParse->iCacheLevel>0 ){ - iReg = sqlite3ExprCodeTarget(pParse, pExpr, target); - }else{ - iReg = ++pParse->nMem; - sqlite3ExprCode(pParse, pExpr, iReg); - pParse->aAlias[iAlias-1] = iReg; - } - } - return iReg; -#else - UNUSED_PARAMETER(iAlias); - return sqlite3ExprCodeTarget(pParse, pExpr, target); -#endif -} - /* ** Generate code into the current Vdbe to evaluate the given ** expression. Attempt to store the results in register "target". ** Return the register where results are stored. ** @@ -69400,11 +70614,11 @@ case TK_REGISTER: { inReg = pExpr->iTable; break; } case TK_AS: { - inReg = codeAlias(pParse, pExpr->iTable, pExpr->pLeft, target); + inReg = sqlite3ExprCodeTarget(pParse, pExpr->pLeft, target); break; } #ifndef SQLITE_OMIT_CAST case TK_CAST: { /* Expressions of the form: CAST(pLeft AS token) */ @@ -69832,10 +71046,15 @@ testcase( regFree1==0 ); cacheX.op = TK_REGISTER; opCompare.op = TK_EQ; opCompare.pLeft = &cacheX; pTest = &opCompare; + /* Ticket b351d95f9cd5ef17e9d9dbae18f5ca8611190001: + ** The value in regFree1 might get SCopy-ed into the file result. + ** So make sure that the regFree1 register is not reused for other + ** purposes and possibly overwritten. */ + regFree1 = 0; } for(i=0; i<nExpr; i=i+2){ sqlite3ExprCachePush(pParse); if( pX ){ assert( pTest!=0 ); @@ -69925,14 +71144,18 @@ */ SQLITE_PRIVATE int sqlite3ExprCode(Parse *pParse, Expr *pExpr, int target){ int inReg; assert( target>0 && target<=pParse->nMem ); - inReg = sqlite3ExprCodeTarget(pParse, pExpr, target); - assert( pParse->pVdbe || pParse->db->mallocFailed ); - if( inReg!=target && pParse->pVdbe ){ - sqlite3VdbeAddOp2(pParse->pVdbe, OP_SCopy, inReg, target); + if( pExpr && pExpr->op==TK_REGISTER ){ + sqlite3VdbeAddOp2(pParse->pVdbe, OP_Copy, pExpr->iTable, target); + }else{ + inReg = sqlite3ExprCodeTarget(pParse, pExpr, target); + assert( pParse->pVdbe || pParse->db->mallocFailed ); + if( inReg!=target && pParse->pVdbe ){ + sqlite3VdbeAddOp2(pParse->pVdbe, OP_SCopy, inReg, target); + } } return target; } /* @@ -70101,23 +71324,18 @@ ){ struct ExprList_item *pItem; int i, n; assert( pList!=0 ); assert( target>0 ); + assert( pParse->pVdbe!=0 ); /* Never gets this far otherwise */ n = pList->nExpr; for(pItem=pList->a, i=0; i<n; i++, pItem++){ - if( pItem->iAlias ){ - int iReg = codeAlias(pParse, pItem->iAlias, pItem->pExpr, target+i); - Vdbe *v = sqlite3GetVdbe(pParse); - if( iReg!=target+i ){ - sqlite3VdbeAddOp2(v, OP_SCopy, iReg, target+i); - } - }else{ - sqlite3ExprCode(pParse, pItem->pExpr, target+i); - } - if( doHardCopy && !pParse->db->mallocFailed ){ - sqlite3ExprHardCopy(pParse, target, n); + Expr *pExpr = pItem->pExpr; + int inReg = sqlite3ExprCodeTarget(pParse, pExpr, target+i); + if( inReg!=target+i ){ + sqlite3VdbeAddOp2(pParse->pVdbe, doHardCopy ? OP_Copy : OP_SCopy, + inReg, target+i); } } return n; } @@ -71095,10 +72313,15 @@ if( pTrig->pSchema==pTempSchema ){ zWhere = whereOrName(db, zWhere, pTrig->zName); } } } + if( zWhere ){ + char *zNew = sqlite3MPrintf(pParse->db, "type='trigger' AND (%s)", zWhere); + sqlite3DbFree(pParse->db, zWhere); + zWhere = zNew; + } return zWhere; } /* ** Generate code to drop and reload the internal representation of table @@ -71702,11 +72925,12 @@ int iIdxCur; /* Cursor open on index being analyzed */ Vdbe *v; /* The virtual machine being built up */ int i; /* Loop counter */ int topOfLoop; /* The top of the loop */ int endOfLoop; /* The end of the loop */ - int addr; /* The address of an instruction */ + int addr = 0; /* The address of an instruction */ + int jZeroRows = 0; /* Jump from here if number of rows is zero */ int iDb; /* Index of database containing pTab */ int regTabname = iMem++; /* Register containing table name */ int regIdxname = iMem++; /* Register containing index name */ int regSampleno = iMem++; /* Register containing next sample number */ int regCol = iMem++; /* Content of a column analyzed table */ @@ -71721,12 +72945,19 @@ int regLast = iMem++; /* Index of last sample to record */ int regFirst = iMem++; /* Index of first sample to record */ #endif v = sqlite3GetVdbe(pParse); - if( v==0 || NEVER(pTab==0) || pTab->pIndex==0 ){ - /* Do no analysis for tables that have no indices */ + if( v==0 || NEVER(pTab==0) ){ + return; + } + if( pTab->tnum==0 ){ + /* Do not gather statistics on views or virtual tables */ + return; + } + if( memcmp(pTab->zName, "sqlite_", 7)==0 ){ + /* Do not gather statistics on system tables */ return; } assert( sqlite3BtreeHoldsAllMutexes(db) ); iDb = sqlite3SchemaToIndex(db, pTab->pSchema); assert( iDb>=0 ); @@ -71739,10 +72970,11 @@ /* Establish a read-lock on the table at the shared-cache level. */ sqlite3TableLock(pParse, iDb, pTab->tnum, 0, pTab->zName); iIdxCur = pParse->nTab++; + sqlite3VdbeAddOp4(v, OP_String8, 0, regTabname, 0, pTab->zName, 0); for(pIdx=pTab->pIndex; pIdx; pIdx=pIdx->pNext){ int nCol = pIdx->nColumn; KeyInfo *pKey = sqlite3IndexKeyinfo(pParse, pIdx); if( iMem+1+(nCol*2)>pParse->nMem ){ @@ -71753,14 +72985,11 @@ assert( iDb==sqlite3SchemaToIndex(db, pIdx->pSchema) ); sqlite3VdbeAddOp4(v, OP_OpenRead, iIdxCur, pIdx->tnum, iDb, (char *)pKey, P4_KEYINFO_HANDOFF); VdbeComment((v, "%s", pIdx->zName)); - /* Populate the registers containing the table and index names. */ - if( pTab->pIndex==pIdx ){ - sqlite3VdbeAddOp4(v, OP_String8, 0, regTabname, 0, pTab->zName, 0); - } + /* Populate the register containing the index name. */ sqlite3VdbeAddOp4(v, OP_String8, 0, regIdxname, 0, pIdx->zName, 0); #ifdef SQLITE_ENABLE_STAT2 /* If this iteration of the loop is generating code to analyze the @@ -71891,12 +73120,14 @@ ** ** If K==0 then no entry is made into the sqlite_stat1 table. ** If K>0 then it is always the case the D>0 so division by zero ** is never possible. */ - addr = sqlite3VdbeAddOp1(v, OP_IfNot, iMem); sqlite3VdbeAddOp2(v, OP_SCopy, iMem, regSampleno); + if( jZeroRows==0 ){ + jZeroRows = sqlite3VdbeAddOp1(v, OP_IfNot, iMem); + } for(i=0; i<nCol; i++){ sqlite3VdbeAddOp4(v, OP_String8, 0, regTemp, 0, " ", 0); sqlite3VdbeAddOp3(v, OP_Concat, regTemp, regSampleno, regSampleno); sqlite3VdbeAddOp3(v, OP_Add, iMem, iMem+i+1, regTemp); sqlite3VdbeAddOp2(v, OP_AddImm, regTemp, -1); @@ -71906,17 +73137,39 @@ } sqlite3VdbeAddOp4(v, OP_MakeRecord, regTabname, 3, regRec, "aaa", 0); sqlite3VdbeAddOp2(v, OP_NewRowid, iStatCur, regRowid); sqlite3VdbeAddOp3(v, OP_Insert, iStatCur, regRec, regRowid); sqlite3VdbeChangeP5(v, OPFLAG_APPEND); + } + + /* If the table has no indices, create a single sqlite_stat1 entry + ** containing NULL as the index name and the row count as the content. + */ + if( pTab->pIndex==0 ){ + sqlite3VdbeAddOp3(v, OP_OpenRead, iIdxCur, pTab->tnum, iDb); + VdbeComment((v, "%s", pTab->zName)); + sqlite3VdbeAddOp2(v, OP_Count, iIdxCur, regSampleno); + sqlite3VdbeAddOp1(v, OP_Close, iIdxCur); + }else{ + assert( jZeroRows>0 ); + addr = sqlite3VdbeAddOp0(v, OP_Goto); + sqlite3VdbeJumpHere(v, jZeroRows); + } + sqlite3VdbeAddOp2(v, OP_Null, 0, regIdxname); + sqlite3VdbeAddOp4(v, OP_MakeRecord, regTabname, 3, regRec, "aaa", 0); + sqlite3VdbeAddOp2(v, OP_NewRowid, iStatCur, regRowid); + sqlite3VdbeAddOp3(v, OP_Insert, iStatCur, regRec, regRowid); + sqlite3VdbeChangeP5(v, OPFLAG_APPEND); + if( pParse->nMem<regRec ) pParse->nMem = regRec; + if( jZeroRows ){ sqlite3VdbeJumpHere(v, addr); } } /* ** Generate code that will cause the most recent index analysis to -** be laoded into internal hash tables where is can be used. +** be loaded into internal hash tables where is can be used. */ static void loadAnalysis(Parse *pParse, int iDb){ Vdbe *v = sqlite3GetVdbe(pParse); if( v ){ sqlite3VdbeAddOp1(v, OP_LoadAnalysis, iDb); @@ -72042,37 +73295,50 @@ /* ** This callback is invoked once for each index when reading the ** sqlite_stat1 table. ** -** argv[0] = name of the index -** argv[1] = results of analysis - on integer for each column +** argv[0] = name of the table +** argv[1] = name of the index (might be NULL) +** argv[2] = results of analysis - on integer for each column +** +** Entries for which argv[1]==NULL simply record the number of rows in +** the table. */ static int analysisLoader(void *pData, int argc, char **argv, char **NotUsed){ analysisInfo *pInfo = (analysisInfo*)pData; Index *pIndex; - int i, c; + Table *pTable; + int i, c, n; unsigned int v; const char *z; - assert( argc==2 ); + assert( argc==3 ); UNUSED_PARAMETER2(NotUsed, argc); - if( argv==0 || argv[0]==0 || argv[1]==0 ){ + if( argv==0 || argv[0]==0 || argv[2]==0 ){ return 0; } - pIndex = sqlite3FindIndex(pInfo->db, argv[0], pInfo->zDatabase); - if( pIndex==0 ){ + pTable = sqlite3FindTable(pInfo->db, argv[0], pInfo->zDatabase); + if( pTable==0 ){ return 0; } - z = argv[1]; - for(i=0; *z && i<=pIndex->nColumn; i++){ + if( argv[1] ){ + pIndex = sqlite3FindIndex(pInfo->db, argv[1], pInfo->zDatabase); + }else{ + pIndex = 0; + } + n = pIndex ? pIndex->nColumn : 0; + z = argv[2]; + for(i=0; *z && i<=n; i++){ v = 0; while( (c=z[0])>='0' && c<='9' ){ v = v*10 + c - '0'; z++; } + if( i==0 ) pTable->nRowEst = v; + if( pIndex==0 ) break; pIndex->aiRowEst[i] = v; if( *z==' ' ) z++; } return 0; } @@ -72092,10 +73358,11 @@ } } sqlite3DbFree(db, pIdx->aSample); } #else + UNUSED_PARAMETER(db); UNUSED_PARAMETER(pIdx); #endif } /* @@ -72143,11 +73410,11 @@ return SQLITE_ERROR; } /* Load new statistics out of the sqlite_stat1 table */ zSql = sqlite3MPrintf(db, - "SELECT idx, stat FROM %Q.sqlite_stat1", sInfo.zDatabase); + "SELECT tbl, idx, stat FROM %Q.sqlite_stat1", sInfo.zDatabase); if( zSql==0 ){ rc = SQLITE_NOMEM; }else{ rc = sqlite3_exec(db, zSql, analysisLoader, &sInfo, 0); sqlite3DbFree(db, zSql); @@ -72360,13 +73627,12 @@ /* Open the database file. If the btree is successfully opened, use ** it to obtain the database schema. At this point the schema may ** or may not be initialised. */ - rc = sqlite3BtreeFactory(db, zFile, 0, SQLITE_DEFAULT_CACHE_SIZE, - db->openFlags | SQLITE_OPEN_MAIN_DB, - &aNew->pBt); + rc = sqlite3BtreeOpen(zFile, db, &aNew->pBt, 0, + db->openFlags | SQLITE_OPEN_MAIN_DB); db->nDb++; if( rc==SQLITE_CONSTRAINT ){ rc = SQLITE_ERROR; zErrDyn = sqlite3MPrintf(db, "database is already attached"); }else if( rc==SQLITE_OK ){ @@ -72603,11 +73869,12 @@ 0, /* pNext */ detachFunc, /* xFunc */ 0, /* xStep */ 0, /* xFinalize */ "sqlite_detach", /* zName */ - 0 /* pHash */ + 0, /* pHash */ + 0 /* pDestructor */ }; codeAttach(pParse, SQLITE_DETACH, &detach_func, pDbname, 0, 0, pDbname); } /* @@ -72624,11 +73891,12 @@ 0, /* pNext */ attachFunc, /* xFunc */ 0, /* xStep */ 0, /* xFinalize */ "sqlite_attach", /* zName */ - 0 /* pHash */ + 0, /* pHash */ + 0 /* pDestructor */ }; codeAttach(pParse, SQLITE_ATTACH, &attach_func, p, p, pDbname, pKey); } #endif /* SQLITE_OMIT_ATTACH */ @@ -73753,12 +75021,13 @@ ** set to the index of the database that the table or view is to be ** created in. */ iDb = sqlite3TwoPartName(pParse, pName1, pName2, &pName); if( iDb<0 ) return; - if( !OMIT_TEMPDB && isTemp && iDb>1 ){ - /* If creating a temp table, the name may not be qualified */ + if( !OMIT_TEMPDB && isTemp && pName2->n>0 && iDb!=1 ){ + /* If creating a temp table, the name may not be qualified. Unless + ** the database name is "temp" anyway. */ sqlite3ErrorMsg(pParse, "temporary table name must be unqualified"); return; } if( !OMIT_TEMPDB && isTemp ) iDb = 1; @@ -73802,21 +75071,22 @@ ** to an sqlite3_declare_vtab() call. In that case only the column names ** and types will be used, so there is no need to test for namespace ** collisions. */ if( !IN_DECLARE_VTAB ){ + char *zDb = db->aDb[iDb].zName; if( SQLITE_OK!=sqlite3ReadSchema(pParse) ){ goto begin_table_error; } - pTable = sqlite3FindTable(db, zName, db->aDb[iDb].zName); + pTable = sqlite3FindTable(db, zName, zDb); if( pTable ){ if( !noErr ){ sqlite3ErrorMsg(pParse, "table %T already exists", pName); } goto begin_table_error; } - if( sqlite3FindIndex(db, zName, 0)!=0 && (iDb==0 || !db->init.busy) ){ + if( sqlite3FindIndex(db, zName, zDb)!=0 ){ sqlite3ErrorMsg(pParse, "there is already an index named %s", zName); goto begin_table_error; } } @@ -73829,10 +75099,11 @@ } pTable->zName = zName; pTable->iPKey = -1; pTable->pSchema = db->aDb[iDb].pSchema; pTable->nRef = 1; + pTable->nRowEst = 1000000; assert( pParse->pNewTable==0 ); pParse->pNewTable = pTable; /* If this is the magic sqlite_sequence table used by autoincrement, ** then record a pointer to this table in the main database structure @@ -74675,16 +75946,14 @@ sqlite3SelectDelete(db, pSelect); return; } sqlite3StartTable(pParse, pName1, pName2, isTemp, 1, 0, noErr); p = pParse->pNewTable; - if( p==0 ){ + if( p==0 || pParse->nErr ){ sqlite3SelectDelete(db, pSelect); return; } - assert( pParse->nErr==0 ); /* If sqlite3StartTable return non-NULL then - ** there could not have been an error */ sqlite3TwoPartName(pParse, pName1, pName2, &pName); iDb = sqlite3SchemaToIndex(db, p->pSchema); if( sqlite3FixInit(&sFix, pParse, iDb, "view", pName) && sqlite3FixSelect(&sFix, pSelect) ){ @@ -75798,11 +77067,12 @@ */ if( pTblName ){ sqlite3RefillIndex(pParse, pIndex, iMem); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddOp4(v, OP_ParseSchema, iDb, 0, 0, - sqlite3MPrintf(db, "name='%q'", pIndex->zName), P4_DYNAMIC); + sqlite3MPrintf(db, "name='%q' AND type='index'", pIndex->zName), + P4_DYNAMIC); sqlite3VdbeAddOp1(v, OP_Expire, 0); } } /* When adding an index to the list of indices for a table, make @@ -75859,18 +77129,18 @@ ** are based on typical values found in actual indices. */ SQLITE_PRIVATE void sqlite3DefaultRowEst(Index *pIdx){ unsigned *a = pIdx->aiRowEst; int i; + unsigned n; assert( a!=0 ); - a[0] = 1000000; - for(i=pIdx->nColumn; i>=5; i--){ - a[i] = 5; - } - while( i>=1 ){ - a[i] = 11 - i; - i--; + a[0] = pIdx->pTable->nRowEst; + if( a[0]<10 ) a[0] = 10; + n = 10; + for(i=1; i<=pIdx->nColumn; i++){ + a[i] = n; + if( n>5 ) n--; } if( pIdx->onError!=OE_None ){ a[pIdx->nColumn] = 1; } } @@ -75926,11 +77196,11 @@ /* Generate code to remove the index and from the master table */ v = sqlite3GetVdbe(pParse); if( v ){ sqlite3BeginWriteOperation(pParse, 1, iDb); sqlite3NestedParse(pParse, - "DELETE FROM %Q.%s WHERE name=%Q", + "DELETE FROM %Q.%s WHERE name=%Q AND type='index'", db->aDb[iDb].zName, SCHEMA_TABLE(iDb), pIndex->zName ); if( sqlite3FindTable(db, "sqlite_stat1", db->aDb[iDb].zName) ){ sqlite3NestedParse(pParse, @@ -76418,11 +77688,11 @@ SQLITE_OPEN_CREATE | SQLITE_OPEN_EXCLUSIVE | SQLITE_OPEN_DELETEONCLOSE | SQLITE_OPEN_TEMP_DB; - rc = sqlite3BtreeFactory(db, 0, 0, SQLITE_DEFAULT_CACHE_SIZE, flags, &pBt); + rc = sqlite3BtreeOpen(0, db, &pBt, 0, flags); if( rc!=SQLITE_OK ){ sqlite3ErrorMsg(pParse, "unable to open a temporary database " "file for storing temporary tables"); pParse->rc = rc; return 1; @@ -77075,11 +78345,11 @@ ** If the SQLITE_PreferBuiltin flag is set, then search the built-in ** functions even if a prior app-defined function was found. And give ** priority to built-in functions. ** ** Except, if createFlag is true, that means that we are trying to - ** install a new function. Whatever FuncDef structure is returned will + ** install a new function. Whatever FuncDef structure is returned it will ** have fields overwritten with new information appropriate for the ** new function. But the FuncDefs for built-in functions are read-only. ** So we must not search for built-ins when creating a new function. */ if( !createFlag && (pBest==0 || (db->flags & SQLITE_PreferBuiltin)!=0) ){ @@ -78091,11 +79361,11 @@ zBuf = sqlite3_mprintf("%.*f",n,r); if( zBuf==0 ){ sqlite3_result_error_nomem(context); return; } - sqlite3AtoF(zBuf, &r); + sqlite3AtoF(zBuf, &r, sqlite3Strlen30(zBuf), SQLITE_UTF8); sqlite3_free(zBuf); } sqlite3_result_double(context, r); } #endif @@ -79256,14 +80526,14 @@ if( caseSensitive ){ pInfo = (struct compareInfo*)&likeInfoAlt; }else{ pInfo = (struct compareInfo*)&likeInfoNorm; } - sqlite3CreateFunc(db, "like", 2, SQLITE_ANY, pInfo, likeFunc, 0, 0); - sqlite3CreateFunc(db, "like", 3, SQLITE_ANY, pInfo, likeFunc, 0, 0); + sqlite3CreateFunc(db, "like", 2, SQLITE_ANY, pInfo, likeFunc, 0, 0, 0); + sqlite3CreateFunc(db, "like", 3, SQLITE_ANY, pInfo, likeFunc, 0, 0, 0); sqlite3CreateFunc(db, "glob", 2, SQLITE_ANY, - (struct compareInfo*)&globInfo, likeFunc, 0,0); + (struct compareInfo*)&globInfo, likeFunc, 0, 0, 0); setLikeOptFlag(db, "glob", SQLITE_FUNC_LIKE | SQLITE_FUNC_CASE); setLikeOptFlag(db, "like", caseSensitive ? (SQLITE_FUNC_LIKE | SQLITE_FUNC_CASE) : SQLITE_FUNC_LIKE); } @@ -79343,14 +80613,14 @@ FUNCTION(upper, 1, 0, 0, upperFunc ), FUNCTION(lower, 1, 0, 0, lowerFunc ), FUNCTION(coalesce, 1, 0, 0, 0 ), FUNCTION(coalesce, 0, 0, 0, 0 ), /* FUNCTION(coalesce, -1, 0, 0, ifnullFunc ), */ - {-1,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"coalesce",0}, + {-1,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"coalesce",0,0}, FUNCTION(hex, 1, 0, 0, hexFunc ), /* FUNCTION(ifnull, 2, 0, 0, ifnullFunc ), */ - {2,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"ifnull",0}, + {2,SQLITE_UTF8,SQLITE_FUNC_COALESCE,0,0,ifnullFunc,0,0,"ifnull",0,0}, FUNCTION(random, 0, 0, 0, randomFunc ), FUNCTION(randomblob, 1, 0, 0, randomBlob ), FUNCTION(nullif, 2, 0, 1, nullifFunc ), FUNCTION(sqlite_version, 0, 0, 0, versionFunc ), FUNCTION(sqlite_source_id, 0, 0, 0, sourceidFunc ), @@ -79373,11 +80643,11 @@ #endif AGGREGATE(sum, 1, 0, 0, sumStep, sumFinalize ), AGGREGATE(total, 1, 0, 0, sumStep, totalFinalize ), AGGREGATE(avg, 1, 0, 0, sumStep, avgFinalize ), /* AGGREGATE(count, 0, 0, 0, countStep, countFinalize ), */ - {0,SQLITE_UTF8,SQLITE_FUNC_COUNT,0,0,0,countStep,countFinalize,"count",0}, + {0,SQLITE_UTF8,SQLITE_FUNC_COUNT,0,0,0,countStep,countFinalize,"count",0,0}, AGGREGATE(count, 1, 0, 0, countStep, countFinalize ), AGGREGATE(group_concat, 1, 0, 0, groupConcatStep, groupConcatFinalize), AGGREGATE(group_concat, 2, 0, 0, groupConcatStep, groupConcatFinalize), LIKEFUNC(glob, 2, &globInfo, SQLITE_FUNC_LIKE|SQLITE_FUNC_CASE), @@ -79784,11 +81054,11 @@ KeyInfo *pKey = sqlite3IndexKeyinfo(pParse, pIdx); sqlite3VdbeAddOp3(v, OP_OpenRead, iCur, pIdx->tnum, iDb); sqlite3VdbeChangeP4(v, -1, (char*)pKey, P4_KEYINFO_HANDOFF); for(i=0; i<nCol; i++){ - sqlite3VdbeAddOp2(v, OP_SCopy, aiCol[i]+1+regData, regTemp+i); + sqlite3VdbeAddOp2(v, OP_Copy, aiCol[i]+1+regData, regTemp+i); } /* If the parent table is the same as the child table, and we are about ** to increment the constraint-counter (i.e. this is an INSERT operation), ** then check if the row being inserted matches itself. If so, do not @@ -84149,11 +85419,11 @@ */ if( sqlite3StrICmp(zLeft,"journal_size_limit")==0 ){ Pager *pPager = sqlite3BtreePager(pDb->pBt); i64 iLimit = -2; if( zRight ){ - sqlite3Atoi64(zRight, &iLimit); + sqlite3Atoi64(zRight, &iLimit, 1000000, SQLITE_UTF8); if( iLimit<-1 ) iLimit = -1; } iLimit = sqlite3PagerJournalSizeLimit(pPager, iLimit); returnSingleInt(pParse, "journal_size_limit", iLimit); }else @@ -86382,11 +87652,10 @@ addr2 = sqlite3VdbeAddOp0(v, OP_Goto); sqlite3VdbeJumpHere(v, addr1); sqlite3VdbeAddOp1(v, OP_Last, pOrderBy->iECursor); sqlite3VdbeAddOp1(v, OP_Delete, pOrderBy->iECursor); sqlite3VdbeJumpHere(v, addr2); - pSelect->iLimit = 0; } } /* ** Add code to implement the OFFSET @@ -86431,15 +87700,17 @@ sqlite3VdbeAddOp3(v, OP_MakeRecord, iMem, N, r1); sqlite3VdbeAddOp2(v, OP_IdxInsert, iTab, r1); sqlite3ReleaseTempReg(pParse, r1); } +#ifndef SQLITE_OMIT_SUBQUERY /* ** Generate an error message when a SELECT is used within a subexpression ** (example: "a IN (SELECT * FROM table)") but it has more than 1 result -** column. We do this in a subroutine because the error occurs in multiple -** places. +** column. We do this in a subroutine because the error used to occur +** in multiple places. (The error only occurs in one place now, but we +** retain the subroutine to minimize code disruption.) */ static int checkForMultiColumnSelectError( Parse *pParse, /* Parse context. */ SelectDest *pDest, /* Destination of SELECT results */ int nExpr /* Number of result columns returned by SELECT */ @@ -86451,10 +87722,11 @@ return 1; }else{ return 0; } } +#endif /* ** This routine generates the code for the inside of the inner loop ** of a SELECT. ** @@ -86530,14 +87802,10 @@ if( pOrderBy==0 ){ codeOffset(v, p, iContinue); } } - if( checkForMultiColumnSelectError(pParse, pDest, pEList->nExpr) ){ - return; - } - switch( eDest ){ /* In this mode, write each query result to the key of the temporary ** table iParm. */ #ifndef SQLITE_OMIT_COMPOUND_SELECT @@ -86662,15 +87930,15 @@ break; } #endif } - /* Jump to the end of the loop if the LIMIT is reached. + /* Jump to the end of the loop if the LIMIT is reached. Except, if + ** there is a sorter, in which case the sorter has already limited + ** the output for us. */ - if( p->iLimit ){ - assert( pOrderBy==0 ); /* If there is an ORDER BY, the call to - ** pushOntoSorter() would have cleared p->iLimit */ + if( pOrderBy==0 && p->iLimit ){ sqlite3VdbeAddOp3(v, OP_IfZero, p->iLimit, iBreak, -1); } } /* @@ -86801,14 +88069,10 @@ } } sqlite3ReleaseTempReg(pParse, regRow); sqlite3ReleaseTempReg(pParse, regRowid); - /* LIMIT has been implemented by the pushOntoSorter() routine. - */ - assert( p->iLimit==0 ); - /* The bottom of the loop */ sqlite3VdbeResolveLabel(v, addrContinue); sqlite3VdbeAddOp2(v, OP_Next, iTab, addr); sqlite3VdbeResolveLabel(v, addrBreak); @@ -87443,10 +88707,11 @@ /* Create the destination temporary table if necessary */ if( dest.eDest==SRT_EphemTab ){ assert( p->pEList ); sqlite3VdbeAddOp2(v, OP_OpenEphemeral, dest.iParm, p->pEList->nExpr); + sqlite3VdbeChangeP5(v, BTREE_UNORDERED); dest.eDest = SRT_Table; } /* Make sure all SELECTs in the statement have the same number of elements ** in their result sets. @@ -87738,11 +89003,11 @@ ** be sent. ** ** regReturn is the number of the register holding the subroutine ** return address. ** -** If regPrev>0 then it is a the first register in a vector that +** If regPrev>0 then it is the first register in a vector that ** records the previous output. mem[regPrev] is a flag that is false ** if there has been no previous output. If regPrev>0 then code is ** generated to suppress duplicates. pKeyInfo is used for comparing ** keys. ** @@ -88121,11 +89386,10 @@ } /* Separate the left and the right query from one another */ p->pPrior = 0; - pPrior->pRightmost = 0; sqlite3ResolveOrderGroupBy(pParse, p, p->pOrderBy, "ORDER"); if( pPrior->pPrior==0 ){ sqlite3ResolveOrderGroupBy(pParse, pPrior, pPrior->pOrderBy, "ORDER"); } @@ -88435,16 +89699,17 @@ ** (1) The subquery and the outer query do not both use aggregates. ** ** (2) The subquery is not an aggregate or the outer query is not a join. ** ** (3) The subquery is not the right operand of a left outer join -** (Originally ticket #306. Strenghtened by ticket #3300) +** (Originally ticket #306. Strengthened by ticket #3300) ** -** (4) The subquery is not DISTINCT or the outer query is not a join. +** (4) The subquery is not DISTINCT. ** -** (5) The subquery is not DISTINCT or the outer query does not use -** aggregates. +** (**) At one point restrictions (4) and (5) defined a subset of DISTINCT +** sub-queries that were excluded from this optimization. Restriction +** (4) has since been expanded to exclude all DISTINCT subqueries. ** ** (6) The subquery does not use aggregates or the outer query is not ** DISTINCT. ** ** (7) The subquery has a FROM clause. @@ -88460,13 +89725,13 @@ ** (11) The subquery and the outer query do not both have ORDER BY clauses. ** ** (**) Not implemented. Subsumed into restriction (3). Was previously ** a separate restriction deriving from ticket #350. ** -** (13) The subquery and outer query do not both use LIMIT +** (13) The subquery and outer query do not both use LIMIT. ** -** (14) The subquery does not use OFFSET +** (14) The subquery does not use OFFSET. ** ** (15) The outer query is not part of a compound select or the ** subquery does not have a LIMIT clause. ** (See ticket #2339 and ticket [02a8e81d44]). ** @@ -88553,13 +89818,13 @@ if( pSub->pOffset ) return 0; /* Restriction (14) */ if( p->pRightmost && pSub->pLimit ){ return 0; /* Restriction (15) */ } if( pSubSrc->nSrc==0 ) return 0; /* Restriction (7) */ - if( ((pSub->selFlags & SF_Distinct)!=0 || pSub->pLimit) - && (pSrc->nSrc>1 || isAgg) ){ /* Restrictions (4)(5)(8)(9) */ - return 0; + if( pSub->selFlags & SF_Distinct ) return 0; /* Restriction (5) */ + if( pSub->pLimit && (pSrc->nSrc>1 || isAgg) ){ + return 0; /* Restrictions (8)(9) */ } if( (p->selFlags & SF_Distinct)!=0 && subqueryIsAgg ){ return 0; /* Restriction (6) */ } if( p->pOrderBy && pSub->pOrderBy ){ @@ -89404,11 +90669,11 @@ ExprList *pList = pF->pExpr->x.pList; assert( !ExprHasProperty(pF->pExpr, EP_xIsSelect) ); if( pList ){ nArg = pList->nExpr; regAgg = sqlite3GetTempRange(pParse, nArg); - sqlite3ExprCodeExprList(pParse, pList, regAgg, 0); + sqlite3ExprCodeExprList(pParse, pList, regAgg, 1); }else{ nArg = 0; regAgg = 0; } if( pF->iDistinct>=0 ){ @@ -89563,10 +90828,19 @@ /* Begin generating code. */ v = sqlite3GetVdbe(pParse); if( v==0 ) goto select_end; + + /* If writing to memory or generating a set + ** only a single column may be output. + */ +#ifndef SQLITE_OMIT_SUBQUERY + if( checkForMultiColumnSelectError(pParse, pDest, pEList->nExpr) ){ + goto select_end; + } +#endif /* Generate code for all sub-queries in the FROM clause */ #if !defined(SQLITE_OMIT_SUBQUERY) || !defined(SQLITE_OMIT_VIEW) for(i=0; !p->pPrior && i<pTabList->nSrc; i++){ @@ -89637,19 +90911,10 @@ } return multiSelect(pParse, p, pDest); } #endif - /* If writing to memory or generating a set - ** only a single column may be output. - */ -#ifndef SQLITE_OMIT_SUBQUERY - if( checkForMultiColumnSelectError(pParse, pDest, pEList->nExpr) ){ - goto select_end; - } -#endif - /* If possible, rewrite the query to use GROUP BY instead of DISTINCT. ** GROUP BY might use an index, DISTINCT never does. */ assert( p->pGroupBy==0 || (p->selFlags & SF_Aggregate)!=0 ); if( (p->selFlags & (SF_Distinct|SF_Aggregate))==SF_Distinct ){ @@ -89708,10 +90973,11 @@ assert( isAgg || pGroupBy ); distinct = pParse->nTab++; pKeyInfo = keyInfoFromExprList(pParse, p->pEList); sqlite3VdbeAddOp4(v, OP_OpenEphemeral, distinct, 0, 0, (char*)pKeyInfo, P4_KEYINFO_HANDOFF); + sqlite3VdbeChangeP5(v, BTREE_UNORDERED); }else{ distinct = -1; } /* Aggregate and non-aggregate queries are handled differently */ @@ -92183,10 +93449,11 @@ ** be stored. */ assert( v ); ephemTab = pParse->nTab++; sqlite3VdbeAddOp2(v, OP_OpenEphemeral, ephemTab, pTab->nCol+1+(pRowid!=0)); + sqlite3VdbeChangeP5(v, BTREE_UNORDERED); /* fill the ephemeral table */ sqlite3SelectDestInit(&dest, SRT_Table, ephemTab); sqlite3Select(pParse, pSelect, &dest); @@ -92321,10 +93588,14 @@ int nDb; /* Number of attached databases */ if( !db->autoCommit ){ sqlite3SetString(pzErrMsg, db, "cannot VACUUM from within a transaction"); return SQLITE_ERROR; + } + if( db->activeVdbeCnt>1 ){ + sqlite3SetString(pzErrMsg, db,"cannot VACUUM - SQL statements in progress"); + return SQLITE_ERROR; } /* Save the current value of the database flags so that it can be ** restored before returning. Then set the writable-schema flag, and ** disable CHECK and foreign key constraints. */ @@ -92923,11 +94194,11 @@ sqlite3DbFree(db, zStmt); v = sqlite3GetVdbe(pParse); sqlite3ChangeCookie(pParse, iDb); sqlite3VdbeAddOp2(v, OP_Expire, 0, 0); - zWhere = sqlite3MPrintf(db, "name='%q'", pTab->zName); + zWhere = sqlite3MPrintf(db, "name='%q' AND type='table'", pTab->zName); sqlite3VdbeAddOp4(v, OP_ParseSchema, iDb, 1, 0, zWhere, P4_DYNAMIC); sqlite3VdbeAddOp4(v, OP_VCreate, iDb, 0, 0, pTab->zName, sqlite3Strlen30(pTab->zName) + 1); } @@ -93225,11 +94496,11 @@ pParse->pNewTable->aCol = 0; } db->pVTab = 0; }else{ sqlite3Error(db, SQLITE_ERROR, zErr); - sqlite3_free(zErr); + sqlite3DbFree(db, zErr); rc = SQLITE_ERROR; } pParse->declareVtab = 0; if( pParse->pVdbe ){ @@ -94163,15 +95434,16 @@ if( op==TK_REGISTER ){ op = pRight->op2; } if( op==TK_VARIABLE ){ Vdbe *pReprepare = pParse->pReprepare; - pVal = sqlite3VdbeGetValue(pReprepare, pRight->iColumn, SQLITE_AFF_NONE); + int iCol = pRight->iColumn; + pVal = sqlite3VdbeGetValue(pReprepare, iCol, SQLITE_AFF_NONE); if( pVal && sqlite3_value_type(pVal)==SQLITE_TEXT ){ z = (char *)sqlite3_value_text(pVal); } - sqlite3VdbeSetVarmask(pParse->pVdbe, pRight->iColumn); + sqlite3VdbeSetVarmask(pParse->pVdbe, iCol); /* IMP: R-23257-02778 */ assert( pRight->op==TK_VARIABLE || pRight->op==TK_REGISTER ); }else if( op==TK_STRING ){ z = pRight->u.zToken; } if( z ){ @@ -94185,11 +95457,11 @@ pPrefix = sqlite3Expr(db, TK_STRING, z); if( pPrefix ) pPrefix->u.zToken[cnt] = 0; *ppPrefix = pPrefix; if( op==TK_VARIABLE ){ Vdbe *v = pParse->pVdbe; - sqlite3VdbeSetVarmask(v, pRight->iColumn); + sqlite3VdbeSetVarmask(v, pRight->iColumn); /* IMP: R-23257-02778 */ if( *pisComplete && pRight->u.zToken[1] ){ /* If the rhs of the LIKE expression is a variable, and the current ** value of the variable means there is no need to invoke the LIKE ** function, then no OP_Variable will be added to the program. ** This causes problems for the sqlite3_bind_parameter_name() @@ -95049,11 +96321,12 @@ /* ** Required because bestIndex() is called by bestOrClauseIndex() */ static void bestIndex( - Parse*, WhereClause*, struct SrcList_item*, Bitmask, ExprList*, WhereCost*); + Parse*, WhereClause*, struct SrcList_item*, + Bitmask, Bitmask, ExprList*, WhereCost*); /* ** This routine attempts to find an scanning strategy that can be used ** to optimize an 'OR' expression that is part of a WHERE clause. ** @@ -95062,11 +96335,12 @@ */ static void bestOrClauseIndex( Parse *pParse, /* The parsing context */ WhereClause *pWC, /* The WHERE clause */ struct SrcList_item *pSrc, /* The FROM clause term to search */ - Bitmask notReady, /* Mask of cursors that are not available */ + Bitmask notReady, /* Mask of cursors not available for indexing */ + Bitmask notValid, /* Cursors not available for any purpose */ ExprList *pOrderBy, /* The ORDER BY clause */ WhereCost *pCost /* Lowest cost query plan */ ){ #ifndef SQLITE_OMIT_OR_OPTIMIZATION const int iCur = pSrc->iCursor; /* The cursor of the table to be accessed */ @@ -95098,19 +96372,19 @@ WHERETRACE(("... Multi-index OR testing for term %d of %d....\n", (pOrTerm - pOrWC->a), (pTerm - pWC->a) )); if( pOrTerm->eOperator==WO_AND ){ WhereClause *pAndWC = &pOrTerm->u.pAndInfo->wc; - bestIndex(pParse, pAndWC, pSrc, notReady, 0, &sTermCost); + bestIndex(pParse, pAndWC, pSrc, notReady, notValid, 0, &sTermCost); }else if( pOrTerm->leftCursor==iCur ){ WhereClause tempWC; tempWC.pParse = pWC->pParse; tempWC.pMaskSet = pWC->pMaskSet; tempWC.op = TK_AND; tempWC.a = pOrTerm; tempWC.nTerm = 1; - bestIndex(pParse, &tempWC, pSrc, notReady, 0, &sTermCost); + bestIndex(pParse, &tempWC, pSrc, notReady, notValid, 0, &sTermCost); }else{ continue; } rTotal += sTermCost.rCost; nRow += sTermCost.nRow; @@ -95199,11 +96473,11 @@ return; } assert( pParse->nQueryLoop >= (double)1 ); pTable = pSrc->pTab; - nTableRow = pTable->pIndex ? pTable->pIndex->aiRowEst[0] : 1000000; + nTableRow = pTable->nRowEst; logN = estLog(nTableRow); costTempIdx = 2*logN*(nTableRow/pParse->nQueryLoop + 1); if( costTempIdx>=pCost->rCost ){ /* The cost of creating the transient table would be greater than ** doing the full table scan */ @@ -95553,11 +96827,12 @@ */ static void bestVirtualIndex( Parse *pParse, /* The parsing context */ WhereClause *pWC, /* The WHERE clause */ struct SrcList_item *pSrc, /* The FROM clause term to search */ - Bitmask notReady, /* Mask of cursors that are not available */ + Bitmask notReady, /* Mask of cursors not available for index */ + Bitmask notValid, /* Cursors not valid for any purpose */ ExprList *pOrderBy, /* The order by clause */ WhereCost *pCost, /* Lowest cost query plan */ sqlite3_index_info **ppIdxInfo /* Index information passed to xBestIndex */ ){ Table *pTab = pSrc->pTab; @@ -95683,11 +96958,11 @@ pIdxInfo->nOrderBy = nOrderBy; /* Try to find a more efficient access pattern by using multiple indexes ** to optimize an OR expression within the WHERE clause. */ - bestOrClauseIndex(pParse, pWC, pSrc, notReady, pOrderBy, pCost); + bestOrClauseIndex(pParse, pWC, pSrc, notReady, notValid, pOrderBy, pCost); } #endif /* SQLITE_OMIT_VIRTUALTABLE */ /* ** Argument pIdx is a pointer to an index structure that has an array of @@ -95809,11 +97084,11 @@ /* The evalConstExpr() function will have already converted any TK_VARIABLE ** expression involved in an comparison into a TK_REGISTER. */ assert( pExpr->op!=TK_VARIABLE ); if( pExpr->op==TK_REGISTER && pExpr->op2==TK_VARIABLE ){ int iVar = pExpr->iColumn; - sqlite3VdbeSetVarmask(pParse->pVdbe, iVar); + sqlite3VdbeSetVarmask(pParse->pVdbe, iVar); /* IMP: R-23257-02778 */ *pp = sqlite3VdbeGetValue(pParse->pReprepare, iVar, aff); return SQLITE_OK; } return sqlite3ValueFromExpr(pParse->db, pExpr, SQLITE_UTF8, aff, pp); } @@ -95964,11 +97239,12 @@ */ static void bestBtreeIndex( Parse *pParse, /* The parsing context */ WhereClause *pWC, /* The WHERE clause */ struct SrcList_item *pSrc, /* The FROM clause term to search */ - Bitmask notReady, /* Mask of cursors that are not available */ + Bitmask notReady, /* Mask of cursors not available for indexing */ + Bitmask notValid, /* Cursors not available for any purpose */ ExprList *pOrderBy, /* The ORDER BY clause */ WhereCost *pCost /* Lowest cost query plan */ ){ int iCur = pSrc->iCursor; /* The cursor of the table to be accessed */ Index *pProbe; /* An index we are evaluating */ @@ -96006,27 +97282,18 @@ Index *pFirst; /* Any other index on the table */ memset(&sPk, 0, sizeof(Index)); sPk.nColumn = 1; sPk.aiColumn = &aiColumnPk; sPk.aiRowEst = aiRowEstPk; - aiRowEstPk[1] = 1; sPk.onError = OE_Replace; sPk.pTable = pSrc->pTab; + aiRowEstPk[0] = pSrc->pTab->nRowEst; + aiRowEstPk[1] = 1; pFirst = pSrc->pTab->pIndex; if( pSrc->notIndexed==0 ){ sPk.pNext = pFirst; } - /* The aiRowEstPk[0] is an estimate of the total number of rows in the - ** table. Get this information from the ANALYZE information if it is - ** available. If not available, assume the table 1 million rows in size. - */ - if( pFirst ){ - assert( pFirst->aiRowEst!=0 ); /* Allocated together with pFirst */ - aiRowEstPk[0] = pFirst->aiRowEst[0]; - }else{ - aiRowEstPk[0] = 1000000; - } pProbe = &sPk; wsFlagMask = ~( WHERE_COLUMN_IN|WHERE_COLUMN_EQ|WHERE_COLUMN_NULL|WHERE_COLUMN_RANGE ); eqTermMask = WO_EQ|WO_IN; @@ -96235,29 +97502,29 @@ ** of output rows, adjust the nRow value accordingly. This only ** matters if the current index is the least costly, so do not bother ** with this step if we already know this index will not be chosen. ** Also, never reduce the output row count below 2 using this step. ** - ** Do not reduce the output row count if pSrc is the only table that - ** is notReady; if notReady is a power of two. This will be the case - ** when the main sqlite3WhereBegin() loop is scanning for a table with - ** and "optimal" index, and on such a scan the output row count - ** reduction is not valid because it does not update the "pCost->used" - ** bitmap. The notReady bitmap will also be a power of two when we - ** are scanning for the last table in a 64-way join. We are willing - ** to bypass this optimization in that corner case. + ** It is critical that the notValid mask be used here instead of + ** the notReady mask. When computing an "optimal" index, the notReady + ** mask will only have one bit set - the bit for the current table. + ** The notValid mask, on the other hand, always has all bits set for + ** tables that are not in outer loops. If notReady is used here instead + ** of notValid, then a optimal index that depends on inner joins loops + ** might be selected even when there exists an optimal index that has + ** no such dependency. */ - if( nRow>2 && cost<=pCost->rCost && (notReady & (notReady-1))!=0 ){ + if( nRow>2 && cost<=pCost->rCost ){ int k; /* Loop counter */ int nSkipEq = nEq; /* Number of == constraints to skip */ int nSkipRange = nBound; /* Number of < constraints to skip */ Bitmask thisTab; /* Bitmap for pSrc */ thisTab = getMask(pWC->pMaskSet, iCur); for(pTerm=pWC->a, k=pWC->nTerm; nRow>2 && k; k--, pTerm++){ if( pTerm->wtFlags & TERM_VIRTUAL ) continue; - if( (pTerm->prereqAll & notReady)!=thisTab ) continue; + if( (pTerm->prereqAll & notValid)!=thisTab ) continue; if( pTerm->eOperator & (WO_EQ|WO_IN|WO_ISNULL) ){ if( nSkipEq ){ /* Ignore the first nEq equality matches since the index ** has already accounted for these */ nSkipEq--; @@ -96335,11 +97602,11 @@ WHERETRACE(("best index is: %s\n", ((pCost->plan.wsFlags & WHERE_NOT_FULLSCAN)==0 ? "none" : pCost->plan.u.pIdx ? pCost->plan.u.pIdx->zName : "ipk") )); - bestOrClauseIndex(pParse, pWC, pSrc, notReady, pOrderBy, pCost); + bestOrClauseIndex(pParse, pWC, pSrc, notReady, notValid, pOrderBy, pCost); bestAutomaticIndex(pParse, pWC, pSrc, notReady, pCost); pCost->plan.wsFlags |= eqTermMask; } /* @@ -96350,26 +97617,27 @@ */ static void bestIndex( Parse *pParse, /* The parsing context */ WhereClause *pWC, /* The WHERE clause */ struct SrcList_item *pSrc, /* The FROM clause term to search */ - Bitmask notReady, /* Mask of cursors that are not available */ + Bitmask notReady, /* Mask of cursors not available for indexing */ + Bitmask notValid, /* Cursors not available for any purpose */ ExprList *pOrderBy, /* The ORDER BY clause */ WhereCost *pCost /* Lowest cost query plan */ ){ #ifndef SQLITE_OMIT_VIRTUALTABLE if( IsVirtual(pSrc->pTab) ){ sqlite3_index_info *p = 0; - bestVirtualIndex(pParse, pWC, pSrc, notReady, pOrderBy, pCost, &p); + bestVirtualIndex(pParse, pWC, pSrc, notReady, notValid, pOrderBy, pCost,&p); if( p->needToFreeIdxStr ){ sqlite3_free(p->idxStr); } sqlite3DbFree(pParse->db, p); }else #endif { - bestBtreeIndex(pParse, pWC, pSrc, notReady, pOrderBy, pCost); + bestBtreeIndex(pParse, pWC, pSrc, notReady, notValid, pOrderBy, pCost); } } /* ** Disable a term in the WHERE clause. Except, do not disable the term @@ -96859,39 +98127,39 @@ ** ** This case is also used when there are no WHERE clause ** constraints but an index is selected anyway, in order ** to force the output order to conform to an ORDER BY. */ - int aStartOp[] = { + static const u8 aStartOp[] = { 0, 0, OP_Rewind, /* 2: (!start_constraints && startEq && !bRev) */ OP_Last, /* 3: (!start_constraints && startEq && bRev) */ OP_SeekGt, /* 4: (start_constraints && !startEq && !bRev) */ OP_SeekLt, /* 5: (start_constraints && !startEq && bRev) */ OP_SeekGe, /* 6: (start_constraints && startEq && !bRev) */ OP_SeekLe /* 7: (start_constraints && startEq && bRev) */ }; - int aEndOp[] = { + static const u8 aEndOp[] = { OP_Noop, /* 0: (!end_constraints) */ OP_IdxGE, /* 1: (end_constraints && !bRev) */ OP_IdxLT /* 2: (end_constraints && bRev) */ }; - int nEq = pLevel->plan.nEq; + int nEq = pLevel->plan.nEq; /* Number of == or IN terms */ int isMinQuery = 0; /* If this is an optimized SELECT min(x).. */ int regBase; /* Base register holding constraint values */ int r1; /* Temp register */ WhereTerm *pRangeStart = 0; /* Inequality constraint at range start */ WhereTerm *pRangeEnd = 0; /* Inequality constraint at range end */ int startEq; /* True if range start uses ==, >= or <= */ int endEq; /* True if range end uses ==, >= or <= */ int start_constraints; /* Start of range is constrained */ int nConstraint; /* Number of constraint terms */ - Index *pIdx; /* The index we will be using */ - int iIdxCur; /* The VDBE cursor for the index */ - int nExtraReg = 0; /* Number of extra registers needed */ - int op; /* Instruction opcode */ + Index *pIdx; /* The index we will be using */ + int iIdxCur; /* The VDBE cursor for the index */ + int nExtraReg = 0; /* Number of extra registers needed */ + int op; /* Instruction opcode */ char *zStartAff; /* Affinity for start of range constraint */ char *zEndAff; /* Affinity for end of range constraint */ pIdx = pLevel->plan.u.pIdx; iIdxCur = pLevel->iIdxCur; @@ -97580,14 +98848,20 @@ ** by waiting for other tables to run first. This "optimal" test works ** by first assuming that the FROM clause is on the inner loop and finding ** its query plan, then checking to see if that query plan uses any ** other FROM clause terms that are notReady. If no notReady terms are ** used then the "optimal" query plan works. + ** + ** Note that the WhereCost.nRow parameter for an optimal scan might + ** not be as small as it would be if the table really were the innermost + ** join. The nRow value can be reduced by WHERE clause constraints + ** that do not use indices. But this nRow reduction only happens if the + ** table really is the innermost join. ** ** The second loop iteration is only performed if no optimal scan - ** strategies were found by the first loop. This 2nd iteration is used to - ** search for the lowest cost scan overall. + ** strategies were found by the first iteration. This second iteration + ** is used to search for the lowest cost scan overall. ** ** Previous versions of SQLite performed only the second iteration - ** the next outermost loop was always that with the lowest overall ** cost. However, this meant that SQLite could select the wrong plan ** for scripts such as the following: @@ -97596,18 +98870,18 @@ ** CREATE TABLE t2(c, d); ** SELECT * FROM t2, t1 WHERE t2.rowid = t1.a; ** ** The best strategy is to iterate through table t1 first. However it ** is not possible to determine this with a simple greedy algorithm. - ** However, since the cost of a linear scan through table t2 is the same + ** Since the cost of a linear scan through table t2 is the same ** as the cost of a linear scan through table t1, a simple greedy ** algorithm may choose to use t2 for the outer loop, which is a much ** costlier approach. */ nUnconstrained = 0; notIndexed = 0; - for(isOptimal=(iFrom<nTabList-1); isOptimal>=0; isOptimal--){ + for(isOptimal=(iFrom<nTabList-1); isOptimal>=0 && bestJ<0; isOptimal--){ Bitmask mask; /* Mask of tables not yet ready */ for(j=iFrom, pTabItem=&pTabList->a[j]; j<nTabList; j++, pTabItem++){ int doNotReorder; /* True if this table should not be reordered */ WhereCost sCost; /* Cost information from best[Virtual]Index() */ ExprList *pOrderBy; /* ORDER BY clause for index to optimize */ @@ -97625,15 +98899,17 @@ assert( pTabItem->pTab ); #ifndef SQLITE_OMIT_VIRTUALTABLE if( IsVirtual(pTabItem->pTab) ){ sqlite3_index_info **pp = &pWInfo->a[j].pIdxInfo; - bestVirtualIndex(pParse, pWC, pTabItem, mask, pOrderBy, &sCost, pp); + bestVirtualIndex(pParse, pWC, pTabItem, mask, notReady, pOrderBy, + &sCost, pp); }else #endif { - bestBtreeIndex(pParse, pWC, pTabItem, mask, pOrderBy, &sCost); + bestBtreeIndex(pParse, pWC, pTabItem, mask, notReady, pOrderBy, + &sCost); } assert( isOptimal || (sCost.used¬Ready)==0 ); /* If an INDEXED BY clause is present, then the plan must use that ** index if it uses any index at all */ @@ -102665,19 +103941,37 @@ /************** End of sqliteicu.h *******************************************/ /************** Continuing where we left off in main.c ***********************/ #endif -/* -** The version of the library -*/ #ifndef SQLITE_AMALGAMATION +/* IMPLEMENTATION-OF: R-46656-45156 The sqlite3_version[] string constant +** contains the text of SQLITE_VERSION macro. +*/ SQLITE_API const char sqlite3_version[] = SQLITE_VERSION; #endif + +/* IMPLEMENTATION-OF: R-53536-42575 The sqlite3_libversion() function returns +** a pointer to the to the sqlite3_version[] string constant. +*/ SQLITE_API const char *sqlite3_libversion(void){ return sqlite3_version; } + +/* IMPLEMENTATION-OF: R-63124-39300 The sqlite3_sourceid() function returns a +** pointer to a string constant whose value is the same as the +** SQLITE_SOURCE_ID C preprocessor macro. +*/ SQLITE_API const char *sqlite3_sourceid(void){ return SQLITE_SOURCE_ID; } + +/* IMPLEMENTATION-OF: R-35210-63508 The sqlite3_libversion_number() function +** returns an integer equal to SQLITE_VERSION_NUMBER. +*/ SQLITE_API int sqlite3_libversion_number(void){ return SQLITE_VERSION_NUMBER; } + +/* IMPLEMENTATION-OF: R-54823-41343 The sqlite3_threadsafe() function returns +** zero if and only if SQLite was compiled mutexing code omitted due to +** the SQLITE_THREADSAFE compile-time option being set to 0. +*/ SQLITE_API int sqlite3_threadsafe(void){ return SQLITE_THREADSAFE; } #if !defined(SQLITE_OMIT_TRACE) && defined(SQLITE_ENABLE_IOTRACE) /* ** If the following function pointer is not NULL and if @@ -102794,10 +104088,17 @@ /* Do the rest of the initialization under the recursive mutex so ** that we will be able to handle recursive calls into ** sqlite3_initialize(). The recursive calls normally come through ** sqlite3_os_init() when it invokes sqlite3_vfs_register(), but other ** recursive calls might also be possible. + ** + ** IMPLEMENTATION-OF: R-00140-37445 SQLite automatically serializes calls + ** to the xInit method, so the xInit method need not be threadsafe. + ** + ** The following mutex is what serializes access to the appdef pcache xInit + ** methods. The sqlite3_pcache_methods.xInit() all is embedded in the + ** call to sqlite3PcacheInitialize(). */ sqlite3_mutex_enter(sqlite3GlobalConfig.pInitMutex); if( sqlite3GlobalConfig.isInit==0 && sqlite3GlobalConfig.inProgress==0 ){ FuncDefHash *pHash = &GLOBAL(FuncDefHash, sqlite3GlobalFunctions); sqlite3GlobalConfig.inProgress = 1; @@ -103074,16 +104375,16 @@ if( cnt<0 ) cnt = 0; if( sz==0 || cnt==0 ){ sz = 0; pStart = 0; }else if( pBuf==0 ){ - sz = ROUND8(sz); + sz = ROUNDDOWN8(sz); /* IMP: R-33038-09382 */ sqlite3BeginBenignMalloc(); - pStart = sqlite3Malloc( sz*cnt ); + pStart = sqlite3Malloc( sz*cnt ); /* IMP: R-61949-35727 */ sqlite3EndBenignMalloc(); }else{ - sz = ROUNDDOWN8(sz); + sz = ROUNDDOWN8(sz); /* IMP: R-33038-09382 */ pStart = pBuf; } db->lookaside.pStart = pStart; db->lookaside.pFree = 0; db->lookaside.sz = (u16)sz; @@ -103122,18 +104423,18 @@ va_list ap; int rc; va_start(ap, op); switch( op ){ case SQLITE_DBCONFIG_LOOKASIDE: { - void *pBuf = va_arg(ap, void*); - int sz = va_arg(ap, int); - int cnt = va_arg(ap, int); + void *pBuf = va_arg(ap, void*); /* IMP: R-21112-12275 */ + int sz = va_arg(ap, int); /* IMP: R-47871-25994 */ + int cnt = va_arg(ap, int); /* IMP: R-04460-53386 */ rc = setupLookaside(db, pBuf, sz, cnt); break; } default: { - rc = SQLITE_ERROR; + rc = SQLITE_ERROR; /* IMP: R-42790-23372 */ break; } } va_end(ap); return rc; @@ -103233,16 +104534,33 @@ } db->nSavepoint = 0; db->nStatement = 0; db->isTransactionSavepoint = 0; } + +/* +** Invoke the destructor function associated with FuncDef p, if any. Except, +** if this is not the last copy of the function, do not invoke it. Multiple +** copies of a single function are created when create_function() is called +** with SQLITE_ANY as the encoding. +*/ +static void functionDestroy(sqlite3 *db, FuncDef *p){ + FuncDestructor *pDestructor = p->pDestructor; + if( pDestructor ){ + pDestructor->nRef--; + if( pDestructor->nRef==0 ){ + pDestructor->xDestroy(pDestructor->pUserData); + sqlite3DbFree(db, pDestructor); + } + } +} /* ** Close an existing SQLite database */ SQLITE_API int sqlite3_close(sqlite3 *db){ - HashElem *i; + HashElem *i; /* Hash table iterator */ int j; if( !db ){ return SQLITE_OK; } @@ -103306,10 +104624,11 @@ for(j=0; j<ArraySize(db->aFunc.a); j++){ FuncDef *pNext, *pHash, *p; for(p=db->aFunc.a[j]; p; p=pHash){ pHash = p->pHash; while( p ){ + functionDestroy(db, p); pNext = p->pNext; sqlite3DbFree(db, p); p = pNext; } } @@ -103580,11 +104899,12 @@ int nArg, int enc, void *pUserData, void (*xFunc)(sqlite3_context*,int,sqlite3_value **), void (*xStep)(sqlite3_context*,int,sqlite3_value **), - void (*xFinal)(sqlite3_context*) + void (*xFinal)(sqlite3_context*), + FuncDestructor *pDestructor ){ FuncDef *p; int nName; assert( sqlite3_mutex_held(db->mutex) ); @@ -103608,14 +104928,14 @@ if( enc==SQLITE_UTF16 ){ enc = SQLITE_UTF16NATIVE; }else if( enc==SQLITE_ANY ){ int rc; rc = sqlite3CreateFunc(db, zFunctionName, nArg, SQLITE_UTF8, - pUserData, xFunc, xStep, xFinal); + pUserData, xFunc, xStep, xFinal, pDestructor); if( rc==SQLITE_OK ){ rc = sqlite3CreateFunc(db, zFunctionName, nArg, SQLITE_UTF16LE, - pUserData, xFunc, xStep, xFinal); + pUserData, xFunc, xStep, xFinal, pDestructor); } if( rc!=SQLITE_OK ){ return rc; } enc = SQLITE_UTF16BE; @@ -103644,10 +104964,19 @@ p = sqlite3FindFunction(db, zFunctionName, nName, nArg, (u8)enc, 1); assert(p || db->mallocFailed); if( !p ){ return SQLITE_NOMEM; } + + /* If an older version of the function with a configured destructor is + ** being replaced invoke the destructor function here. */ + functionDestroy(db, p); + + if( pDestructor ){ + pDestructor->nRef++; + } + p->pDestructor = pDestructor; p->flags = 0; p->xFunc = xFunc; p->xStep = xStep; p->xFinalize = xFinal; p->pUserData = pUserData; @@ -103658,21 +104987,53 @@ /* ** Create new user functions. */ SQLITE_API int sqlite3_create_function( sqlite3 *db, - const char *zFunctionName, + const char *zFunc, int nArg, int enc, void *p, void (*xFunc)(sqlite3_context*,int,sqlite3_value **), void (*xStep)(sqlite3_context*,int,sqlite3_value **), void (*xFinal)(sqlite3_context*) ){ - int rc; + return sqlite3_create_function_v2(db, zFunc, nArg, enc, p, xFunc, xStep, + xFinal, 0); +} + +SQLITE_API int sqlite3_create_function_v2( + sqlite3 *db, + const char *zFunc, + int nArg, + int enc, + void *p, + void (*xFunc)(sqlite3_context*,int,sqlite3_value **), + void (*xStep)(sqlite3_context*,int,sqlite3_value **), + void (*xFinal)(sqlite3_context*), + void (*xDestroy)(void *) +){ + int rc = SQLITE_ERROR; + FuncDestructor *pArg = 0; sqlite3_mutex_enter(db->mutex); - rc = sqlite3CreateFunc(db, zFunctionName, nArg, enc, p, xFunc, xStep, xFinal); + if( xDestroy ){ + pArg = (FuncDestructor *)sqlite3DbMallocZero(db, sizeof(FuncDestructor)); + if( !pArg ){ + xDestroy(p); + goto out; + } + pArg->xDestroy = xDestroy; + pArg->pUserData = p; + } + rc = sqlite3CreateFunc(db, zFunc, nArg, enc, p, xFunc, xStep, xFinal, pArg); + if( pArg && pArg->nRef==0 ){ + assert( rc!=SQLITE_OK ); + xDestroy(p); + sqlite3DbFree(db, pArg); + } + + out: rc = sqlite3ApiExit(db, rc); sqlite3_mutex_leave(db->mutex); return rc; } @@ -103690,11 +105051,11 @@ int rc; char *zFunc8; sqlite3_mutex_enter(db->mutex); assert( !db->mallocFailed ); zFunc8 = sqlite3Utf16to8(db, zFunctionName, -1, SQLITE_UTF16NATIVE); - rc = sqlite3CreateFunc(db, zFunc8, nArg, eTextRep, p, xFunc, xStep, xFinal); + rc = sqlite3CreateFunc(db, zFunc8, nArg, eTextRep, p, xFunc, xStep, xFinal,0); sqlite3DbFree(db, zFunc8); rc = sqlite3ApiExit(db, rc); sqlite3_mutex_leave(db->mutex); return rc; } @@ -103721,11 +105082,11 @@ int nName = sqlite3Strlen30(zName); int rc; sqlite3_mutex_enter(db->mutex); if( sqlite3FindFunction(db, zName, nName, nArg, SQLITE_UTF8, 0)==0 ){ sqlite3CreateFunc(db, zName, nArg, SQLITE_UTF8, - 0, sqlite3InvalidFunction, 0, 0); + 0, sqlite3InvalidFunction, 0, 0, 0); } rc = sqlite3ApiExit(db, SQLITE_OK); sqlite3_mutex_leave(db->mutex); return rc; } @@ -103859,11 +105220,14 @@ ** registered using sqlite3_wal_hook(). Likewise, registering a callback ** using sqlite3_wal_hook() disables the automatic checkpoint mechanism ** configured by this function. */ SQLITE_API int sqlite3_wal_autocheckpoint(sqlite3 *db, int nFrame){ -#ifndef SQLITE_OMIT_WAL +#ifdef SQLITE_OMIT_WAL + UNUSED_PARAMETER(db); + UNUSED_PARAMETER(nFrame); +#else if( nFrame>0 ){ sqlite3_wal_hook(db, sqlite3WalDefaultHook, SQLITE_INT_TO_PTR(nFrame)); }else{ sqlite3_wal_hook(db, 0, 0); } @@ -103989,64 +105353,10 @@ #if SQLITE_TEMP_STORE<1 || SQLITE_TEMP_STORE>3 return 0; #endif } -/* -** This routine is called to create a connection to a database BTree -** driver. If zFilename is the name of a file, then that file is -** opened and used. If zFilename is the magic name ":memory:" then -** the database is stored in memory (and is thus forgotten as soon as -** the connection is closed.) If zFilename is NULL then the database -** is a "virtual" database for transient use only and is deleted as -** soon as the connection is closed. -** -** A virtual database can be either a disk file (that is automatically -** deleted when the file is closed) or it an be held entirely in memory. -** The sqlite3TempInMemory() function is used to determine which. -*/ -SQLITE_PRIVATE int sqlite3BtreeFactory( - sqlite3 *db, /* Main database when opening aux otherwise 0 */ - const char *zFilename, /* Name of the file containing the BTree database */ - int omitJournal, /* if TRUE then do not journal this file */ - int nCache, /* How many pages in the page cache */ - int vfsFlags, /* Flags passed through to vfsOpen */ - Btree **ppBtree /* Pointer to new Btree object written here */ -){ - int btFlags = 0; - int rc; - - assert( sqlite3_mutex_held(db->mutex) ); - assert( ppBtree != 0); - if( omitJournal ){ - btFlags |= BTREE_OMIT_JOURNAL; - } - if( db->flags & SQLITE_NoReadlock ){ - btFlags |= BTREE_NO_READLOCK; - } -#ifndef SQLITE_OMIT_MEMORYDB - if( zFilename==0 && sqlite3TempInMemory(db) ){ - zFilename = ":memory:"; - } -#endif - - if( (vfsFlags & SQLITE_OPEN_MAIN_DB)!=0 && (zFilename==0 || *zFilename==0) ){ - vfsFlags = (vfsFlags & ~SQLITE_OPEN_MAIN_DB) | SQLITE_OPEN_TEMP_DB; - } - rc = sqlite3BtreeOpen(zFilename, (sqlite3 *)db, ppBtree, btFlags, vfsFlags); - - /* If the B-Tree was successfully opened, set the pager-cache size to the - ** default value. Except, if the call to BtreeOpen() returned a handle - ** open on an existing shared pager-cache, do not change the pager-cache - ** size. - */ - if( rc==SQLITE_OK && 0==sqlite3BtreeSchema(*ppBtree, 0, 0) ){ - sqlite3BtreeSetCacheSize(*ppBtree, nCache); - } - return rc; -} - /* ** Return UTF-8 encoded English language explanation of the most recent ** error. */ SQLITE_API const char *sqlite3_errmsg(sqlite3 *db){ @@ -104285,21 +105595,43 @@ ** It merely prevents new constructs that exceed the limit ** from forming. */ SQLITE_API int sqlite3_limit(sqlite3 *db, int limitId, int newLimit){ int oldLimit; + + + /* EVIDENCE-OF: R-30189-54097 For each limit category SQLITE_LIMIT_NAME + ** there is a hard upper bound set at compile-time by a C preprocessor + ** macro called SQLITE_MAX_NAME. (The "_LIMIT_" in the name is changed to + ** "_MAX_".) + */ + assert( aHardLimit[SQLITE_LIMIT_LENGTH]==SQLITE_MAX_LENGTH ); + assert( aHardLimit[SQLITE_LIMIT_SQL_LENGTH]==SQLITE_MAX_SQL_LENGTH ); + assert( aHardLimit[SQLITE_LIMIT_COLUMN]==SQLITE_MAX_COLUMN ); + assert( aHardLimit[SQLITE_LIMIT_EXPR_DEPTH]==SQLITE_MAX_EXPR_DEPTH ); + assert( aHardLimit[SQLITE_LIMIT_COMPOUND_SELECT]==SQLITE_MAX_COMPOUND_SELECT); + assert( aHardLimit[SQLITE_LIMIT_VDBE_OP]==SQLITE_MAX_VDBE_OP ); + assert( aHardLimit[SQLITE_LIMIT_FUNCTION_ARG]==SQLITE_MAX_FUNCTION_ARG ); + assert( aHardLimit[SQLITE_LIMIT_ATTACHED]==SQLITE_MAX_ATTACHED ); + assert( aHardLimit[SQLITE_LIMIT_LIKE_PATTERN_LENGTH]== + SQLITE_MAX_LIKE_PATTERN_LENGTH ); + assert( aHardLimit[SQLITE_LIMIT_VARIABLE_NUMBER]==SQLITE_MAX_VARIABLE_NUMBER); + assert( aHardLimit[SQLITE_LIMIT_TRIGGER_DEPTH]==SQLITE_MAX_TRIGGER_DEPTH ); + assert( SQLITE_LIMIT_TRIGGER_DEPTH==(SQLITE_N_LIMIT-1) ); + + if( limitId<0 || limitId>=SQLITE_N_LIMIT ){ return -1; } oldLimit = db->aLimit[limitId]; - if( newLimit>=0 ){ + if( newLimit>=0 ){ /* IMP: R-52476-28732 */ if( newLimit>aHardLimit[limitId] ){ - newLimit = aHardLimit[limitId]; + newLimit = aHardLimit[limitId]; /* IMP: R-51463-25634 */ } db->aLimit[limitId] = newLimit; } - return oldLimit; + return oldLimit; /* IMP: R-53341-35419 */ } /* ** This routine does the work of opening a database on behalf of ** sqlite3_open() and sqlite3_open16(). The database filename "zFilename" @@ -104318,10 +105650,28 @@ *ppDb = 0; #ifndef SQLITE_OMIT_AUTOINIT rc = sqlite3_initialize(); if( rc ) return rc; #endif + + /* Only allow sensible combinations of bits in the flags argument. + ** Throw an error if any non-sense combination is used. If we + ** do not block illegal combinations here, it could trigger + ** assert() statements in deeper layers. Sensible combinations + ** are: + ** + ** 1: SQLITE_OPEN_READONLY + ** 2: SQLITE_OPEN_READWRITE + ** 6: SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE + */ + assert( SQLITE_OPEN_READONLY == 0x01 ); + assert( SQLITE_OPEN_READWRITE == 0x02 ); + assert( SQLITE_OPEN_CREATE == 0x04 ); + testcase( (1<<(flags&7))==0x02 ); /* READONLY */ + testcase( (1<<(flags&7))==0x04 ); /* READWRITE */ + testcase( (1<<(flags&7))==0x40 ); /* READWRITE | CREATE */ + if( ((1<<(flags&7)) & 0x46)==0 ) return SQLITE_MISUSE; if( sqlite3GlobalConfig.bCoreMutex==0 ){ isThreadsafe = 0; }else if( flags & SQLITE_OPEN_NOMUTEX ){ isThreadsafe = 0; @@ -104352,11 +105702,12 @@ SQLITE_OPEN_MAIN_JOURNAL | SQLITE_OPEN_TEMP_JOURNAL | SQLITE_OPEN_SUBJOURNAL | SQLITE_OPEN_MASTER_JOURNAL | SQLITE_OPEN_NOMUTEX | - SQLITE_OPEN_FULLMUTEX + SQLITE_OPEN_FULLMUTEX | + SQLITE_OPEN_WAL ); /* Allocate the sqlite data structure */ db = sqlite3MallocZero( sizeof(sqlite3) ); if( db==0 ) goto opendb_out; @@ -104424,13 +105775,12 @@ createCollation(db, "NOCASE", SQLITE_UTF8, SQLITE_COLL_NOCASE, 0, nocaseCollatingFunc, 0); /* Open the backend database driver */ db->openFlags = flags; - rc = sqlite3BtreeFactory(db, zFilename, 0, SQLITE_DEFAULT_CACHE_SIZE, - flags | SQLITE_OPEN_MAIN_DB, - &db->aDb[0].pBt); + rc = sqlite3BtreeOpen(zFilename, db, &db->aDb[0].pBt, 0, + flags | SQLITE_OPEN_MAIN_DB); if( rc!=SQLITE_OK ){ if( rc==SQLITE_IOERR_NOMEM ){ rc = SQLITE_NOMEM; } sqlite3Error(db, rc, 0); @@ -105132,10 +106482,26 @@ */ case SQLITE_TESTCTRL_PGHDRSZ: { rc = sizeof(PgHdr); break; } + + /* sqlite3_test_control(SQLITE_TESTCTRL_SCRATCHMALLOC, sz, &pNew, pFree); + ** + ** Pass pFree into sqlite3ScratchFree(). + ** If sz>0 then allocate a scratch buffer into pNew. + */ + case SQLITE_TESTCTRL_SCRATCHMALLOC: { + void *pFree, **ppNew; + int sz; + sz = va_arg(ap, int); + ppNew = va_arg(ap, void**); + pFree = va_arg(ap, void*); + if( sz ) *ppNew = sqlite3ScratchMalloc(sz); + sqlite3ScratchFree(pFree); + break; + } } va_end(ap); #endif /* SQLITE_OMIT_BUILTIN_TEST */ return rc; @@ -106321,10 +107687,11 @@ ); SQLITE_PRIVATE int sqlite3Fts3ReadBlock(Fts3Table*, sqlite3_int64, char const**, int*); SQLITE_PRIVATE int sqlite3Fts3AllSegdirs(Fts3Table*, sqlite3_stmt **); SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeLocal(Fts3Cursor*, u32*); SQLITE_PRIVATE int sqlite3Fts3MatchinfoDocsizeGlobal(Fts3Cursor*, u32*); +SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *); /* Flags allowed as part of the 4th argument to SegmentReaderIterate() */ #define FTS3_SEGMENT_REQUIRE_POS 0x00000001 #define FTS3_SEGMENT_IGNORE_EMPTY 0x00000002 #define FTS3_SEGMENT_COLUMN_FILTER 0x00000004 @@ -108270,10 +109637,13 @@ zQuery); } return rc; } + rc = sqlite3Fts3ReadLock(p); + if( rc!=SQLITE_OK ) return rc; + rc = evalFts3Expr(p, pCsr->pExpr, &pCsr->aDoclist, &pCsr->nDoclist, 0); pCsr->pNextId = pCsr->aDoclist; pCsr->iPrevId = 0; } @@ -108648,15 +110018,18 @@ static int fts3RenameMethod( sqlite3_vtab *pVtab, /* Virtual table handle */ const char *zName /* New name of table */ ){ Fts3Table *p = (Fts3Table *)pVtab; - sqlite3 *db; /* Database connection */ + sqlite3 *db = p->db; /* Database connection */ int rc; /* Return Code */ - - db = p->db; - rc = SQLITE_OK; + + rc = sqlite3Fts3PendingTermsFlush(p); + if( rc!=SQLITE_OK ){ + return rc; + } + fts3DbExec(&rc, db, "ALTER TABLE %Q.'%q_content' RENAME TO '%q_content';", p->zDb, p->zName, zName ); if( rc==SQLITE_ERROR ) rc = SQLITE_OK; @@ -108720,11 +110093,11 @@ ** in files fts3_tokenizer1.c and fts3_porter.c respectively. The following ** two forward declarations are for functions declared in these files ** used to retrieve the respective implementations. ** ** Calling sqlite3Fts3SimpleTokenizerModule() sets the value pointed -** to by the argument to point a the "simple" tokenizer implementation. +** to by the argument to point to the "simple" tokenizer implementation. ** Function ...PorterTokenizerModule() sets *pModule to point to the ** porter tokenizer/stemmer implementation. */ SQLITE_PRIVATE void sqlite3Fts3SimpleTokenizerModule(sqlite3_tokenizer_module const**ppModule); SQLITE_PRIVATE void sqlite3Fts3PorterTokenizerModule(sqlite3_tokenizer_module const**ppModule); @@ -108922,11 +110295,11 @@ ** is defined to accept an argument of type char, and always returns 0 for ** any values that fall outside of the range of the unsigned char type (i.e. ** negative values). */ static int fts3isspace(char c){ - return (c&0x80)==0 ? isspace(c) : 0; + return c==' ' || c=='\t' || c=='\n' || c=='\r' || c=='\v' || c=='\f'; } /* ** Extract the next token from buffer z (length n) using the tokenizer ** and other information (column names etc.) in pParse. Create an Fts3Expr @@ -111309,10 +112682,13 @@ static int simpleDelim(simple_tokenizer *t, unsigned char c){ return c<0x80 && t->delim[c]; } +static int fts3_isalnum(int x){ + return (x>='0' && x<='9') || (x>='A' && x<='Z') || (x>='a' && x<='z'); +} /* ** Create a new tokenizer instance. */ static int simpleCreate( @@ -111343,11 +112719,11 @@ } } else { /* Mark non-alphanumeric ASCII characters as delimiters */ int i; for(i=1; i<0x80; i++){ - t->delim[i] = !isalnum(i) ? -1 : 0; + t->delim[i] = !fts3_isalnum(i) ? -1 : 0; } } *ppTokenizer = &t->base; return SQLITE_OK; @@ -111449,11 +112825,11 @@ for(i=0; i<n; i++){ /* TODO(shess) This needs expansion to handle UTF-8 ** case-insensitivity. */ unsigned char ch = p[iStartOffset+i]; - c->pToken[i] = (char)(ch<0x80 ? tolower(ch) : ch); + c->pToken[i] = (char)((ch>='A' && ch<='Z') ? ch-'A'+'a' : ch); } *ppToken = c->pToken; *pnBytes = n; *piStartOffset = iStartOffset; *piEndOffset = c->iOffset; @@ -111808,10 +113184,40 @@ return SQLITE_CORRUPT; } } return SQLITE_OK; } + +/* +** This function ensures that the caller has obtained a shared-cache +** table-lock on the %_content table. This is required before reading +** data from the fts3 table. If this lock is not acquired first, then +** the caller may end up holding read-locks on the %_segments and %_segdir +** tables, but no read-lock on the %_content table. If this happens +** a second connection will be able to write to the fts3 table, but +** attempting to commit those writes might return SQLITE_LOCKED or +** SQLITE_LOCKED_SHAREDCACHE (because the commit attempts to obtain +** write-locks on the %_segments and %_segdir ** tables). +** +** We try to avoid this because if FTS3 returns any error when committing +** a transaction, the whole transaction will be rolled back. And this is +** not what users expect when they get SQLITE_LOCKED_SHAREDCACHE. It can +** still happen if the user reads data directly from the %_segments or +** %_segdir tables instead of going through FTS3 though. +*/ +SQLITE_PRIVATE int sqlite3Fts3ReadLock(Fts3Table *p){ + int rc; /* Return code */ + sqlite3_stmt *pStmt; /* Statement used to obtain lock */ + + rc = fts3SqlStmt(p, SQL_SELECT_CONTENT_BY_ROWID, &pStmt, 0); + if( rc==SQLITE_OK ){ + sqlite3_bind_null(pStmt, 1); + sqlite3_step(pStmt); + rc = sqlite3_reset(pStmt); + } + return rc; +} /* ** Set *ppStmt to a statement handle that may be used to iterate through ** all rows in the %_segdir table, from oldest to newest. If successful, ** return SQLITE_OK. If an error occurs while preparing the statement, @@ -115299,10 +116705,49 @@ ** ************************************************************************* ** This file contains code for implementations of the r-tree and r*-tree ** algorithms packaged as an SQLite virtual table module. */ + +/* +** Database Format of R-Tree Tables +** -------------------------------- +** +** The data structure for a single virtual r-tree table is stored in three +** native SQLite tables declared as follows. In each case, the '%' character +** in the table name is replaced with the user-supplied name of the r-tree +** table. +** +** CREATE TABLE %_node(nodeno INTEGER PRIMARY KEY, data BLOB) +** CREATE TABLE %_parent(nodeno INTEGER PRIMARY KEY, parentnode INTEGER) +** CREATE TABLE %_rowid(rowid INTEGER PRIMARY KEY, nodeno INTEGER) +** +** The data for each node of the r-tree structure is stored in the %_node +** table. For each node that is not the root node of the r-tree, there is +** an entry in the %_parent table associating the node with its parent. +** And for each row of data in the table, there is an entry in the %_rowid +** table that maps from the entries rowid to the id of the node that it +** is stored on. +** +** The root node of an r-tree always exists, even if the r-tree table is +** empty. The nodeno of the root node is always 1. All other nodes in the +** table must be the same size as the root node. The content of each node +** is formatted as follows: +** +** 1. If the node is the root node (node 1), then the first 2 bytes +** of the node contain the tree depth as a big-endian integer. +** For non-root nodes, the first 2 bytes are left unused. +** +** 2. The next 2 bytes contain the number of entries currently +** stored in the node. +** +** 3. The remainder of the node contains the node entries. Each entry +** consists of a single 8-byte integer followed by an even number +** of 4-byte coordinates. For leaf nodes the integer is the rowid +** of a record. For internal nodes it is the node number of a +** child page. +*/ #if !defined(SQLITE_CORE) || defined(SQLITE_ENABLE_RTREE) /* ** This file contains an implementation of a couple of different variants @@ -115340,18 +116785,22 @@ #endif #if VARIANT_RSTARTREE_SPLIT #define AssignCells splitNodeStartree #endif +#if !defined(NDEBUG) && !defined(SQLITE_DEBUG) +# define NDEBUG 1 +#endif #ifndef SQLITE_CORE SQLITE_EXTENSION_INIT1 #else #endif #ifndef SQLITE_AMALGAMATION +#include "sqlite3rtree.h" typedef sqlite3_int64 i64; typedef unsigned char u8; typedef unsigned int u32; #endif @@ -115358,10 +116807,12 @@ typedef struct Rtree Rtree; typedef struct RtreeCursor RtreeCursor; typedef struct RtreeNode RtreeNode; typedef struct RtreeCell RtreeCell; typedef struct RtreeConstraint RtreeConstraint; +typedef struct RtreeMatchArg RtreeMatchArg; +typedef struct RtreeGeomCallback RtreeGeomCallback; typedef union RtreeCoord RtreeCoord; /* The rtree may have between 1 and RTREE_MAX_DIMENSIONS dimensions. */ #define RTREE_MAX_DIMENSIONS 5 @@ -115427,10 +116878,19 @@ */ #define RTREE_MINCELLS(p) ((((p)->iNodeSize-4)/(p)->nBytesPerCell)/3) #define RTREE_REINSERT(p) RTREE_MINCELLS(p) #define RTREE_MAXCELLS 51 +/* +** The smallest possible node-size is (512-64)==448 bytes. And the largest +** supported cell size is 48 bytes (8 byte rowid + ten 4 byte coordinates). +** Therefore all non-root nodes must contain at least 3 entries. Since +** 2^40 is greater than 2^64, an r-tree structure always has a depth of +** 40 or less. +*/ +#define RTREE_MAX_DEPTH 40 + /* ** An rtree cursor object. */ struct RtreeCursor { sqlite3_vtab_cursor base; @@ -115459,39 +116919,27 @@ /* ** A search constraint. */ struct RtreeConstraint { - int iCoord; /* Index of constrained coordinate */ - int op; /* Constraining operation */ - double rValue; /* Constraint value. */ + int iCoord; /* Index of constrained coordinate */ + int op; /* Constraining operation */ + double rValue; /* Constraint value. */ + int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *); + sqlite3_rtree_geometry *pGeom; /* Constraint callback argument for a MATCH */ }; /* Possible values for RtreeConstraint.op */ -#define RTREE_EQ 0x41 -#define RTREE_LE 0x42 -#define RTREE_LT 0x43 -#define RTREE_GE 0x44 -#define RTREE_GT 0x45 +#define RTREE_EQ 0x41 +#define RTREE_LE 0x42 +#define RTREE_LT 0x43 +#define RTREE_GE 0x44 +#define RTREE_GT 0x45 +#define RTREE_MATCH 0x46 /* ** An rtree structure node. -** -** Data format (RtreeNode.zData): -** -** 1. If the node is the root node (node 1), then the first 2 bytes -** of the node contain the tree depth as a big-endian integer. -** For non-root nodes, the first 2 bytes are left unused. -** -** 2. The next 2 bytes contain the number of entries currently -** stored in the node. -** -** 3. The remainder of the node contains the node entries. Each entry -** consists of a single 8-byte integer followed by an even number -** of 4-byte coordinates. For leaf nodes the integer is the rowid -** of a record. For internal nodes it is the node number of a -** child page. */ struct RtreeNode { RtreeNode *pParent; /* Parent node */ i64 iNode; int nRef; @@ -115507,10 +116955,44 @@ struct RtreeCell { i64 iRowid; RtreeCoord aCoord[RTREE_MAX_DIMENSIONS*2]; }; + +/* +** Value for the first field of every RtreeMatchArg object. The MATCH +** operator tests that the first field of a blob operand matches this +** value to avoid operating on invalid blobs (which could cause a segfault). +*/ +#define RTREE_GEOMETRY_MAGIC 0x891245AB + +/* +** An instance of this structure must be supplied as a blob argument to +** the right-hand-side of an SQL MATCH operator used to constrain an +** r-tree query. +*/ +struct RtreeMatchArg { + u32 magic; /* Always RTREE_GEOMETRY_MAGIC */ + int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *); + void *pContext; + int nParam; + double aParam[1]; +}; + +/* +** When a geometry callback is created (see sqlite3_rtree_geometry_callback), +** a single instance of the following structure is allocated. It is used +** as the context for the user-function created by by s_r_g_c(). The object +** is eventually deleted by the destructor mechanism provided by +** sqlite3_create_function_v2() (which is called by s_r_g_c() to create +** the geometry callback function). +*/ +struct RtreeGeomCallback { + int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *); + void *pContext; +}; + #ifndef MAX # define MAX(x,y) ((x) < (y) ? (y) : (x)) #endif #ifndef MIN # define MIN(x,y) ((x) > (y) ? (y) : (x)) @@ -115589,14 +117071,12 @@ /* ** Clear the content of node p (set all bytes to 0x00). */ static void nodeZero(Rtree *pRtree, RtreeNode *p){ - if( p ){ - memset(&p->zData[2], 0, pRtree->iNodeSize-2); - p->isDirty = 1; - } + memset(&p->zData[2], 0, pRtree->iNodeSize-2); + p->isDirty = 1; } /* ** Given a node number iNode, return the corresponding key to use ** in the Rtree.aHash table. @@ -115612,26 +117092,23 @@ ** Search the node hash table for node iNode. If found, return a pointer ** to it. Otherwise, return 0. */ static RtreeNode *nodeHashLookup(Rtree *pRtree, i64 iNode){ RtreeNode *p; - assert( iNode!=0 ); for(p=pRtree->aHash[nodeHash(iNode)]; p && p->iNode!=iNode; p=p->pNext); return p; } /* ** Add node pNode to the node hash table. */ static void nodeHashInsert(Rtree *pRtree, RtreeNode *pNode){ - if( pNode ){ - int iHash; - assert( pNode->pNext==0 ); - iHash = nodeHash(pNode->iNode); - pNode->pNext = pRtree->aHash[iHash]; - pRtree->aHash[iHash] = pNode; - } + int iHash; + assert( pNode->pNext==0 ); + iHash = nodeHash(pNode->iNode); + pNode->pNext = pRtree->aHash[iHash]; + pRtree->aHash[iHash] = pNode; } /* ** Remove node pNode from the node hash table. */ @@ -115649,15 +117126,15 @@ ** Allocate and return new r-tree node. Initially, (RtreeNode.iNode==0), ** indicating that node has not yet been assigned a node number. It is ** assigned a node number when nodeWrite() is called to write the ** node contents out to the database. */ -static RtreeNode *nodeNew(Rtree *pRtree, RtreeNode *pParent, int zero){ +static RtreeNode *nodeNew(Rtree *pRtree, RtreeNode *pParent){ RtreeNode *pNode; pNode = (RtreeNode *)sqlite3_malloc(sizeof(RtreeNode) + pRtree->iNodeSize); if( pNode ){ - memset(pNode, 0, sizeof(RtreeNode) + (zero?pRtree->iNodeSize:0)); + memset(pNode, 0, sizeof(RtreeNode) + pRtree->iNodeSize); pNode->zData = (u8 *)&pNode[1]; pNode->nRef = 1; pNode->pParent = pParent; pNode->isDirty = 1; nodeReference(pParent); @@ -115674,10 +117151,11 @@ i64 iNode, /* Node number to load */ RtreeNode *pParent, /* Either the parent node or NULL */ RtreeNode **ppNode /* OUT: Acquired node */ ){ int rc; + int rc2 = SQLITE_OK; RtreeNode *pNode; /* Check if the requested node is already in the hash table. If so, ** increase its reference count and return it. */ @@ -115690,43 +117168,67 @@ pNode->nRef++; *ppNode = pNode; return SQLITE_OK; } - pNode = (RtreeNode *)sqlite3_malloc(sizeof(RtreeNode) + pRtree->iNodeSize); - if( !pNode ){ - *ppNode = 0; - return SQLITE_NOMEM; - } - pNode->pParent = pParent; - pNode->zData = (u8 *)&pNode[1]; - pNode->nRef = 1; - pNode->iNode = iNode; - pNode->isDirty = 0; - pNode->pNext = 0; - sqlite3_bind_int64(pRtree->pReadNode, 1, iNode); rc = sqlite3_step(pRtree->pReadNode); if( rc==SQLITE_ROW ){ const u8 *zBlob = sqlite3_column_blob(pRtree->pReadNode, 0); - assert( sqlite3_column_bytes(pRtree->pReadNode, 0)==pRtree->iNodeSize ); - memcpy(pNode->zData, zBlob, pRtree->iNodeSize); - nodeReference(pParent); + if( pRtree->iNodeSize==sqlite3_column_bytes(pRtree->pReadNode, 0) ){ + pNode = (RtreeNode *)sqlite3_malloc(sizeof(RtreeNode)+pRtree->iNodeSize); + if( !pNode ){ + rc2 = SQLITE_NOMEM; + }else{ + pNode->pParent = pParent; + pNode->zData = (u8 *)&pNode[1]; + pNode->nRef = 1; + pNode->iNode = iNode; + pNode->isDirty = 0; + pNode->pNext = 0; + memcpy(pNode->zData, zBlob, pRtree->iNodeSize); + nodeReference(pParent); + } + } + } + rc = sqlite3_reset(pRtree->pReadNode); + if( rc==SQLITE_OK ) rc = rc2; + + /* If the root node was just loaded, set pRtree->iDepth to the height + ** of the r-tree structure. A height of zero means all data is stored on + ** the root node. A height of one means the children of the root node + ** are the leaves, and so on. If the depth as specified on the root node + ** is greater than RTREE_MAX_DEPTH, the r-tree structure must be corrupt. + */ + if( pNode && iNode==1 ){ + pRtree->iDepth = readInt16(pNode->zData); + if( pRtree->iDepth>RTREE_MAX_DEPTH ){ + rc = SQLITE_CORRUPT; + } + } + + /* If no error has occurred so far, check if the "number of entries" + ** field on the node is too large. If so, set the return code to + ** SQLITE_CORRUPT. + */ + if( pNode && rc==SQLITE_OK ){ + if( NCELL(pNode)>((pRtree->iNodeSize-4)/pRtree->nBytesPerCell) ){ + rc = SQLITE_CORRUPT; + } + } + + if( rc==SQLITE_OK ){ + if( pNode!=0 ){ + nodeHashInsert(pRtree, pNode); + }else{ + rc = SQLITE_CORRUPT; + } + *ppNode = pNode; }else{ sqlite3_free(pNode); - pNode = 0; - } - - *ppNode = pNode; - rc = sqlite3_reset(pRtree->pReadNode); - - if( rc==SQLITE_OK && iNode==1 ){ - pRtree->iDepth = readInt16(pNode->zData); - } - - assert( (rc==SQLITE_OK && pNode) || (pNode==0 && rc!=SQLITE_OK) ); - nodeHashInsert(pRtree, pNode); + *ppNode = 0; + } return rc; } /* @@ -115775,12 +117277,11 @@ int nMaxCell; /* Maximum number of cells for pNode */ nMaxCell = (pRtree->iNodeSize-4)/pRtree->nBytesPerCell; nCell = NCELL(pNode); - assert(nCell<=nMaxCell); - + assert( nCell<=nMaxCell ); if( nCell<nMaxCell ){ nodeOverwriteCell(pRtree, pNode, pCell, nCell); writeInt16(&pNode->zData[2], nCell+1); pNode->isDirty = 1; } @@ -115996,18 +117497,37 @@ *ppCursor = (sqlite3_vtab_cursor *)pCsr; return rc; } + +/* +** Free the RtreeCursor.aConstraint[] array and its contents. +*/ +static void freeCursorConstraints(RtreeCursor *pCsr){ + if( pCsr->aConstraint ){ + int i; /* Used to iterate through constraint array */ + for(i=0; i<pCsr->nConstraint; i++){ + sqlite3_rtree_geometry *pGeom = pCsr->aConstraint[i].pGeom; + if( pGeom ){ + if( pGeom->xDelUser ) pGeom->xDelUser(pGeom->pUser); + sqlite3_free(pGeom); + } + } + sqlite3_free(pCsr->aConstraint); + pCsr->aConstraint = 0; + } +} + /* ** Rtree virtual table module xClose method. */ static int rtreeClose(sqlite3_vtab_cursor *cur){ Rtree *pRtree = (Rtree *)(cur->pVtab); int rc; RtreeCursor *pCsr = (RtreeCursor *)cur; - sqlite3_free(pCsr->aConstraint); + freeCursorConstraints(pCsr); rc = nodeRelease(pRtree, pCsr->pNode); sqlite3_free(pCsr); return rc; } @@ -116019,18 +117539,44 @@ */ static int rtreeEof(sqlite3_vtab_cursor *cur){ RtreeCursor *pCsr = (RtreeCursor *)cur; return (pCsr->pNode==0); } + +/* +** The r-tree constraint passed as the second argument to this function is +** guaranteed to be a MATCH constraint. +*/ +static int testRtreeGeom( + Rtree *pRtree, /* R-Tree object */ + RtreeConstraint *pConstraint, /* MATCH constraint to test */ + RtreeCell *pCell, /* Cell to test */ + int *pbRes /* OUT: Test result */ +){ + int i; + double aCoord[RTREE_MAX_DIMENSIONS*2]; + int nCoord = pRtree->nDim*2; + + assert( pConstraint->op==RTREE_MATCH ); + assert( pConstraint->pGeom ); + + for(i=0; i<nCoord; i++){ + aCoord[i] = DCOORD(pCell->aCoord[i]); + } + return pConstraint->xGeom(pConstraint->pGeom, nCoord, aCoord, pbRes); +} /* ** Cursor pCursor currently points to a cell in a non-leaf page. -** Return true if the sub-tree headed by the cell is filtered +** Set *pbEof to true if the sub-tree headed by the cell is filtered ** (excluded) by the constraints in the pCursor->aConstraint[] ** array, or false otherwise. +** +** Return SQLITE_OK if successful or an SQLite error code if an error +** occurs within a geometry callback. */ -static int testRtreeCell(Rtree *pRtree, RtreeCursor *pCursor){ +static int testRtreeCell(Rtree *pRtree, RtreeCursor *pCursor, int *pbEof){ RtreeCell cell; int ii; int bRes = 0; nodeGetCell(pRtree, pCursor->pNode, pCursor->iCell, &cell); @@ -116038,56 +117584,92 @@ RtreeConstraint *p = &pCursor->aConstraint[ii]; double cell_min = DCOORD(cell.aCoord[(p->iCoord>>1)*2]); double cell_max = DCOORD(cell.aCoord[(p->iCoord>>1)*2+1]); assert(p->op==RTREE_LE || p->op==RTREE_LT || p->op==RTREE_GE - || p->op==RTREE_GT || p->op==RTREE_EQ + || p->op==RTREE_GT || p->op==RTREE_EQ || p->op==RTREE_MATCH ); switch( p->op ){ - case RTREE_LE: case RTREE_LT: bRes = p->rValue<cell_min; break; - case RTREE_GE: case RTREE_GT: bRes = p->rValue>cell_max; break; - case RTREE_EQ: + case RTREE_LE: case RTREE_LT: + bRes = p->rValue<cell_min; + break; + + case RTREE_GE: case RTREE_GT: + bRes = p->rValue>cell_max; + break; + + case RTREE_EQ: bRes = (p->rValue>cell_max || p->rValue<cell_min); break; + + default: { + int rc; + assert( p->op==RTREE_MATCH ); + rc = testRtreeGeom(pRtree, p, &cell, &bRes); + if( rc!=SQLITE_OK ){ + return rc; + } + bRes = !bRes; + break; + } } } - return bRes; + *pbEof = bRes; + return SQLITE_OK; } /* -** Return true if the cell that cursor pCursor currently points to +** Test if the cell that cursor pCursor currently points to ** would be filtered (excluded) by the constraints in the -** pCursor->aConstraint[] array, or false otherwise. +** pCursor->aConstraint[] array. If so, set *pbEof to true before +** returning. If the cell is not filtered (excluded) by the constraints, +** set pbEof to zero. +** +** Return SQLITE_OK if successful or an SQLite error code if an error +** occurs within a geometry callback. ** ** This function assumes that the cell is part of a leaf node. */ -static int testRtreeEntry(Rtree *pRtree, RtreeCursor *pCursor){ +static int testRtreeEntry(Rtree *pRtree, RtreeCursor *pCursor, int *pbEof){ RtreeCell cell; int ii; + *pbEof = 0; nodeGetCell(pRtree, pCursor->pNode, pCursor->iCell, &cell); for(ii=0; ii<pCursor->nConstraint; ii++){ RtreeConstraint *p = &pCursor->aConstraint[ii]; double coord = DCOORD(cell.aCoord[p->iCoord]); int res; assert(p->op==RTREE_LE || p->op==RTREE_LT || p->op==RTREE_GE - || p->op==RTREE_GT || p->op==RTREE_EQ + || p->op==RTREE_GT || p->op==RTREE_EQ || p->op==RTREE_MATCH ); switch( p->op ){ case RTREE_LE: res = (coord<=p->rValue); break; case RTREE_LT: res = (coord<p->rValue); break; case RTREE_GE: res = (coord>=p->rValue); break; case RTREE_GT: res = (coord>p->rValue); break; case RTREE_EQ: res = (coord==p->rValue); break; + default: { + int rc; + assert( p->op==RTREE_MATCH ); + rc = testRtreeGeom(pRtree, p, &cell, &res); + if( rc!=SQLITE_OK ){ + return rc; + } + break; + } } - if( !res ) return 1; + if( !res ){ + *pbEof = 1; + return SQLITE_OK; + } } - return 0; + return SQLITE_OK; } /* ** Cursor pCursor currently points at a node that heads a sub-tree of ** height iHeight (if iHeight==0, then the node is a leaf). Descend @@ -116110,17 +117692,17 @@ int iSavedCell = pCursor->iCell; assert( iHeight>=0 ); if( iHeight==0 ){ - isEof = testRtreeEntry(pRtree, pCursor); + rc = testRtreeEntry(pRtree, pCursor, &isEof); }else{ - isEof = testRtreeCell(pRtree, pCursor); + rc = testRtreeCell(pRtree, pCursor, &isEof); } - if( isEof || iHeight==0 ){ + if( rc!=SQLITE_OK || isEof || iHeight==0 ){ *pEof = isEof; - return SQLITE_OK; + return rc; } iRowid = nodeGetRowid(pRtree, pCursor->pNode, pCursor->iCell); rc = nodeAcquire(pRtree, iRowid, pCursor->pNode, &pChild); if( rc!=SQLITE_OK ){ @@ -116152,45 +117734,59 @@ /* ** One of the cells in node pNode is guaranteed to have a 64-bit ** integer value equal to iRowid. Return the index of this cell. */ -static int nodeRowidIndex(Rtree *pRtree, RtreeNode *pNode, i64 iRowid){ +static int nodeRowidIndex( + Rtree *pRtree, + RtreeNode *pNode, + i64 iRowid, + int *piIndex +){ int ii; - for(ii=0; nodeGetRowid(pRtree, pNode, ii)!=iRowid; ii++){ - assert( ii<(NCELL(pNode)-1) ); + int nCell = NCELL(pNode); + for(ii=0; ii<nCell; ii++){ + if( nodeGetRowid(pRtree, pNode, ii)==iRowid ){ + *piIndex = ii; + return SQLITE_OK; + } } - return ii; + return SQLITE_CORRUPT; } /* ** Return the index of the cell containing a pointer to node pNode ** in its parent. If pNode is the root node, return -1. */ -static int nodeParentIndex(Rtree *pRtree, RtreeNode *pNode){ +static int nodeParentIndex(Rtree *pRtree, RtreeNode *pNode, int *piIndex){ RtreeNode *pParent = pNode->pParent; if( pParent ){ - return nodeRowidIndex(pRtree, pParent, pNode->iNode); + return nodeRowidIndex(pRtree, pParent, pNode->iNode, piIndex); } - return -1; + *piIndex = -1; + return SQLITE_OK; } /* ** Rtree virtual table module xNext method. */ static int rtreeNext(sqlite3_vtab_cursor *pVtabCursor){ Rtree *pRtree = (Rtree *)(pVtabCursor->pVtab); RtreeCursor *pCsr = (RtreeCursor *)pVtabCursor; int rc = SQLITE_OK; + + /* RtreeCursor.pNode must not be NULL. If is is NULL, then this cursor is + ** already at EOF. It is against the rules to call the xNext() method of + ** a cursor that has already reached EOF. + */ + assert( pCsr->pNode ); if( pCsr->iStrategy==1 ){ /* This "scan" is a direct lookup by rowid. There is no next entry. */ nodeRelease(pRtree, pCsr->pNode); pCsr->pNode = 0; - } - - else if( pCsr->pNode ){ + }else{ /* Move to the next entry that matches the configured constraints. */ int iHeight = 0; while( pCsr->pNode ){ RtreeNode *pNode = pCsr->pNode; int nCell = NCELL(pNode); @@ -116200,11 +117796,14 @@ if( rc!=SQLITE_OK || !isEof ){ return rc; } } pCsr->pNode = pNode->pParent; - pCsr->iCell = nodeParentIndex(pRtree, pNode); + rc = nodeParentIndex(pRtree, pNode, &pCsr->iCell); + if( rc!=SQLITE_OK ){ + return rc; + } nodeReference(pCsr->pNode); nodeRelease(pRtree, pNode); iHeight++; } } @@ -116268,10 +117867,55 @@ rc = sqlite3_reset(pRtree->pReadRowid); } return rc; } +/* +** This function is called to configure the RtreeConstraint object passed +** as the second argument for a MATCH constraint. The value passed as the +** first argument to this function is the right-hand operand to the MATCH +** operator. +*/ +static int deserializeGeometry(sqlite3_value *pValue, RtreeConstraint *pCons){ + RtreeMatchArg *p; + sqlite3_rtree_geometry *pGeom; + int nBlob; + + /* Check that value is actually a blob. */ + if( !sqlite3_value_type(pValue)==SQLITE_BLOB ) return SQLITE_ERROR; + + /* Check that the blob is roughly the right size. */ + nBlob = sqlite3_value_bytes(pValue); + if( nBlob<sizeof(RtreeMatchArg) + || ((nBlob-sizeof(RtreeMatchArg))%sizeof(double))!=0 + ){ + return SQLITE_ERROR; + } + + pGeom = (sqlite3_rtree_geometry *)sqlite3_malloc( + sizeof(sqlite3_rtree_geometry) + nBlob + ); + if( !pGeom ) return SQLITE_NOMEM; + memset(pGeom, 0, sizeof(sqlite3_rtree_geometry)); + p = (RtreeMatchArg *)&pGeom[1]; + + memcpy(p, sqlite3_value_blob(pValue), nBlob); + if( p->magic!=RTREE_GEOMETRY_MAGIC + || nBlob!=(sizeof(RtreeMatchArg) + (p->nParam-1)*sizeof(double)) + ){ + sqlite3_free(pGeom); + return SQLITE_ERROR; + } + + pGeom->pContext = p->pContext; + pGeom->nParam = p->nParam; + pGeom->aParam = p->aParam; + + pCons->xGeom = p->xGeom; + pCons->pGeom = pGeom; + return SQLITE_OK; +} /* ** Rtree virtual table module xFilter method. */ static int rtreeFilter( @@ -116286,22 +117930,22 @@ int ii; int rc = SQLITE_OK; rtreeReference(pRtree); - sqlite3_free(pCsr->aConstraint); - pCsr->aConstraint = 0; + freeCursorConstraints(pCsr); pCsr->iStrategy = idxNum; if( idxNum==1 ){ /* Special case - lookup by rowid. */ RtreeNode *pLeaf; /* Leaf on which the required cell resides */ i64 iRowid = sqlite3_value_int64(argv[0]); rc = findLeafNode(pRtree, iRowid, &pLeaf); pCsr->pNode = pLeaf; - if( pLeaf && rc==SQLITE_OK ){ - pCsr->iCell = nodeRowidIndex(pRtree, pLeaf, iRowid); + if( pLeaf ){ + assert( rc==SQLITE_OK ); + rc = nodeRowidIndex(pRtree, pLeaf, iRowid, &pCsr->iCell); } }else{ /* Normal case - r-tree scan. Set up the RtreeCursor.aConstraint array ** with the configured constraints. */ @@ -116309,16 +117953,28 @@ pCsr->aConstraint = sqlite3_malloc(sizeof(RtreeConstraint)*argc); pCsr->nConstraint = argc; if( !pCsr->aConstraint ){ rc = SQLITE_NOMEM; }else{ + memset(pCsr->aConstraint, 0, sizeof(RtreeConstraint)*argc); assert( (idxStr==0 && argc==0) || strlen(idxStr)==argc*2 ); for(ii=0; ii<argc; ii++){ RtreeConstraint *p = &pCsr->aConstraint[ii]; p->op = idxStr[ii*2]; p->iCoord = idxStr[ii*2+1]-'a'; - p->rValue = sqlite3_value_double(argv[ii]); + if( p->op==RTREE_MATCH ){ + /* A MATCH operator. The right-hand-side must be a blob that + ** can be cast into an RtreeMatchArg object. One created using + ** an sqlite3_rtree_geometry_callback() SQL user function. + */ + rc = deserializeGeometry(argv[ii], p); + if( rc!=SQLITE_OK ){ + break; + } + }else{ + p->rValue = sqlite3_value_double(argv[ii]); + } } } } if( rc==SQLITE_OK ){ @@ -116355,15 +118011,14 @@ ** least desirable): ** ** idxNum idxStr Strategy ** ------------------------------------------------ ** 1 Unused Direct lookup by rowid. -** 2 See below R-tree query. -** 3 Unused Full table scan. +** 2 See below R-tree query or full-table scan. ** ------------------------------------------------ ** -** If strategy 1 or 3 is used, then idxStr is not meaningful. If strategy +** If strategy 1 is used, then idxStr is not meaningful. If strategy ** 2 is used, idxStr is formatted to contain 2 bytes for each ** constraint used. The first two bytes of idxStr correspond to ** the constraint in sqlite3_index_info.aConstraintUsage[] with ** (argvIndex==1) etc. ** @@ -116375,10 +118030,11 @@ ** = 0x41 ('A') ** <= 0x42 ('B') ** < 0x43 ('C') ** >= 0x44 ('D') ** > 0x45 ('E') +** MATCH 0x46 ('F') ** ---------------------- ** ** The second of each pair of bytes identifies the coordinate column ** to which the constraint applies. The leftmost coordinate column ** is 'a', the second from the left 'b' etc. @@ -116413,43 +118069,47 @@ */ pIdxInfo->estimatedCost = 10.0; return SQLITE_OK; } - if( p->usable && p->iColumn>0 ){ + if( p->usable && (p->iColumn>0 || p->op==SQLITE_INDEX_CONSTRAINT_MATCH) ){ + int j, opmsk; + static const unsigned char compatible[] = { 0, 0, 1, 1, 2, 2 }; u8 op = 0; switch( p->op ){ case SQLITE_INDEX_CONSTRAINT_EQ: op = RTREE_EQ; break; case SQLITE_INDEX_CONSTRAINT_GT: op = RTREE_GT; break; case SQLITE_INDEX_CONSTRAINT_LE: op = RTREE_LE; break; case SQLITE_INDEX_CONSTRAINT_LT: op = RTREE_LT; break; case SQLITE_INDEX_CONSTRAINT_GE: op = RTREE_GE; break; - } - if( op ){ - /* Make sure this particular constraint has not been used before. - ** If it has been used before, ignore it. - ** - ** A <= or < can be used if there is a prior >= or >. - ** A >= or > can be used if there is a prior < or <=. - ** A <= or < is disqualified if there is a prior <=, <, or ==. - ** A >= or > is disqualified if there is a prior >=, >, or ==. - ** A == is disqualifed if there is any prior constraint. - */ - int j, opmsk; - static const unsigned char compatible[] = { 0, 0, 1, 1, 2, 2 }; - assert( compatible[RTREE_EQ & 7]==0 ); - assert( compatible[RTREE_LT & 7]==1 ); - assert( compatible[RTREE_LE & 7]==1 ); - assert( compatible[RTREE_GT & 7]==2 ); - assert( compatible[RTREE_GE & 7]==2 ); - cCol = p->iColumn - 1 + 'a'; - opmsk = compatible[op & 7]; - for(j=0; j<iIdx; j+=2){ - if( zIdxStr[j+1]==cCol && (compatible[zIdxStr[j] & 7] & opmsk)!=0 ){ - op = 0; - break; - } + default: + assert( p->op==SQLITE_INDEX_CONSTRAINT_MATCH ); + op = RTREE_MATCH; + break; + } + assert( op!=0 ); + + /* Make sure this particular constraint has not been used before. + ** If it has been used before, ignore it. + ** + ** A <= or < can be used if there is a prior >= or >. + ** A >= or > can be used if there is a prior < or <=. + ** A <= or < is disqualified if there is a prior <=, <, or ==. + ** A >= or > is disqualified if there is a prior >=, >, or ==. + ** A == is disqualifed if there is any prior constraint. + */ + assert( compatible[RTREE_EQ & 7]==0 ); + assert( compatible[RTREE_LT & 7]==1 ); + assert( compatible[RTREE_LE & 7]==1 ); + assert( compatible[RTREE_GT & 7]==2 ); + assert( compatible[RTREE_GE & 7]==2 ); + cCol = p->iColumn - 1 + 'a'; + opmsk = compatible[op & 7]; + for(j=0; j<iIdx; j+=2){ + if( zIdxStr[j+1]==cCol && (compatible[zIdxStr[j] & 7] & opmsk)!=0 ){ + op = 0; + break; } } if( op ){ assert( iIdx<sizeof(zIdxStr)-1 ); zIdxStr[iIdx++] = op; @@ -116553,11 +118213,16 @@ int iExclude ){ int ii; float overlap = 0.0; for(ii=0; ii<nCell; ii++){ - if( ii!=iExclude ){ +#if VARIANT_RSTARTREE_CHOOSESUBTREE + if( ii!=iExclude ) +#else + assert( iExclude==-1 ); +#endif + { int jj; float o = 1.0; for(jj=0; jj<(pRtree->nDim*2); jj+=2){ double x1; double x2; @@ -116646,26 +118311,35 @@ /* Select the child node which will be enlarged the least if pCell ** is inserted into it. Resolve ties by choosing the entry with ** the smallest area. */ for(iCell=0; iCell<nCell; iCell++){ + int bBest = 0; float growth; float area; float overlap = 0.0; nodeGetCell(pRtree, pNode, iCell, &cell); growth = cellGrowth(pRtree, &cell, pCell); area = cellArea(pRtree, &cell); + #if VARIANT_RSTARTREE_CHOOSESUBTREE if( ii==(pRtree->iDepth-1) ){ overlap = cellOverlapEnlargement(pRtree,&cell,pCell,aCell,nCell,iCell); } -#endif if( (iCell==0) || (overlap<fMinOverlap) || (overlap==fMinOverlap && growth<fMinGrowth) || (overlap==fMinOverlap && growth==fMinGrowth && area<fMinArea) ){ + bBest = 1; + } +#else + if( iCell==0||growth<fMinGrowth||(growth==fMinGrowth && area<fMinArea) ){ + bBest = 1; + } +#endif + if( bBest ){ fMinOverlap = overlap; fMinGrowth = growth; fMinArea = area; iBest = cell.iRowid; } @@ -116684,29 +118358,34 @@ /* ** A cell with the same content as pCell has just been inserted into ** the node pNode. This function updates the bounding box cells in ** all ancestor elements. */ -static void AdjustTree( +static int AdjustTree( Rtree *pRtree, /* Rtree table */ RtreeNode *pNode, /* Adjust ancestry of this node. */ RtreeCell *pCell /* This cell was just inserted */ ){ RtreeNode *p = pNode; while( p->pParent ){ - RtreeCell cell; RtreeNode *pParent = p->pParent; - int iCell = nodeParentIndex(pRtree, p); + RtreeCell cell; + int iCell; + + if( nodeParentIndex(pRtree, p, &iCell) ){ + return SQLITE_CORRUPT; + } nodeGetCell(pRtree, pParent, iCell, &cell); if( !cellContains(pRtree, &cell, pCell) ){ cellUnion(pRtree, &cell, pCell); nodeOverwriteCell(pRtree, pParent, &cell, iCell); } p = pParent; } + return SQLITE_OK; } /* ** Write mapping (iRowid->iNode) to the <rtree>_rowid table. */ @@ -117231,18 +118910,18 @@ nodeZero(pRtree, pNode); memcpy(&aCell[nCell], pCell, sizeof(RtreeCell)); nCell++; if( pNode->iNode==1 ){ - pRight = nodeNew(pRtree, pNode, 1); - pLeft = nodeNew(pRtree, pNode, 1); + pRight = nodeNew(pRtree, pNode); + pLeft = nodeNew(pRtree, pNode); pRtree->iDepth++; pNode->isDirty = 1; writeInt16(pNode->zData, pRtree->iDepth); }else{ pLeft = pNode; - pRight = nodeNew(pRtree, pLeft->pParent, 1); + pRight = nodeNew(pRtree, pLeft->pParent); nodeReference(pLeft); } if( !pLeft || !pRight ){ rc = SQLITE_NOMEM; @@ -117255,12 +118934,16 @@ rc = AssignCells(pRtree, aCell, nCell, pLeft, pRight, &leftbbox, &rightbbox); if( rc!=SQLITE_OK ){ goto splitnode_out; } - /* Ensure both child nodes have node numbers assigned to them. */ - if( (0==pRight->iNode && SQLITE_OK!=(rc = nodeWrite(pRtree, pRight))) + /* Ensure both child nodes have node numbers assigned to them by calling + ** nodeWrite(). Node pRight always needs a node number, as it was created + ** by nodeNew() above. But node pLeft sometimes already has a node number. + ** In this case avoid the all to nodeWrite(). + */ + if( SQLITE_OK!=(rc = nodeWrite(pRtree, pRight)) || (0==pLeft->iNode && SQLITE_OK!=(rc = nodeWrite(pRtree, pLeft))) ){ goto splitnode_out; } @@ -117272,13 +118955,19 @@ if( rc!=SQLITE_OK ){ goto splitnode_out; } }else{ RtreeNode *pParent = pLeft->pParent; - int iCell = nodeParentIndex(pRtree, pLeft); - nodeOverwriteCell(pRtree, pParent, &leftbbox, iCell); - AdjustTree(pRtree, pParent, &leftbbox); + int iCell; + rc = nodeParentIndex(pRtree, pLeft, &iCell); + if( rc==SQLITE_OK ){ + nodeOverwriteCell(pRtree, pParent, &leftbbox, iCell); + rc = AdjustTree(pRtree, pParent, &leftbbox); + } + if( rc!=SQLITE_OK ){ + goto splitnode_out; + } } if( (rc = rtreeInsertCell(pRtree, pRight->pParent, &rightbbox, iHeight+1)) ){ goto splitnode_out; } @@ -117318,44 +119007,73 @@ nodeRelease(pRtree, pLeft); sqlite3_free(aCell); return rc; } +/* +** If node pLeaf is not the root of the r-tree and its pParent pointer is +** still NULL, load all ancestor nodes of pLeaf into memory and populate +** the pLeaf->pParent chain all the way up to the root node. +** +** This operation is required when a row is deleted (or updated - an update +** is implemented as a delete followed by an insert). SQLite provides the +** rowid of the row to delete, which can be used to find the leaf on which +** the entry resides (argument pLeaf). Once the leaf is located, this +** function is called to determine its ancestry. +*/ static int fixLeafParent(Rtree *pRtree, RtreeNode *pLeaf){ int rc = SQLITE_OK; - if( pLeaf->iNode!=1 && pLeaf->pParent==0 ){ - sqlite3_bind_int64(pRtree->pReadParent, 1, pLeaf->iNode); - if( sqlite3_step(pRtree->pReadParent)==SQLITE_ROW ){ - i64 iNode = sqlite3_column_int64(pRtree->pReadParent, 0); - rc = nodeAcquire(pRtree, iNode, 0, &pLeaf->pParent); - }else{ - rc = SQLITE_ERROR; - } - sqlite3_reset(pRtree->pReadParent); - if( rc==SQLITE_OK ){ - rc = fixLeafParent(pRtree, pLeaf->pParent); - } + RtreeNode *pChild = pLeaf; + while( rc==SQLITE_OK && pChild->iNode!=1 && pChild->pParent==0 ){ + int rc2 = SQLITE_OK; /* sqlite3_reset() return code */ + sqlite3_bind_int64(pRtree->pReadParent, 1, pChild->iNode); + rc = sqlite3_step(pRtree->pReadParent); + if( rc==SQLITE_ROW ){ + RtreeNode *pTest; /* Used to test for reference loops */ + i64 iNode; /* Node number of parent node */ + + /* Before setting pChild->pParent, test that we are not creating a + ** loop of references (as we would if, say, pChild==pParent). We don't + ** want to do this as it leads to a memory leak when trying to delete + ** the referenced counted node structures. + */ + iNode = sqlite3_column_int64(pRtree->pReadParent, 0); + for(pTest=pLeaf; pTest && pTest->iNode!=iNode; pTest=pTest->pParent); + if( !pTest ){ + rc2 = nodeAcquire(pRtree, iNode, 0, &pChild->pParent); + } + } + rc = sqlite3_reset(pRtree->pReadParent); + if( rc==SQLITE_OK ) rc = rc2; + if( rc==SQLITE_OK && !pChild->pParent ) rc = SQLITE_CORRUPT; + pChild = pChild->pParent; } return rc; } static int deleteCell(Rtree *, RtreeNode *, int, int); static int removeNode(Rtree *pRtree, RtreeNode *pNode, int iHeight){ int rc; + int rc2; RtreeNode *pParent; int iCell; assert( pNode->nRef==1 ); /* Remove the entry in the parent cell. */ - iCell = nodeParentIndex(pRtree, pNode); - pParent = pNode->pParent; - pNode->pParent = 0; - if( SQLITE_OK!=(rc = deleteCell(pRtree, pParent, iCell, iHeight+1)) - || SQLITE_OK!=(rc = nodeRelease(pRtree, pParent)) - ){ + rc = nodeParentIndex(pRtree, pNode, &iCell); + if( rc==SQLITE_OK ){ + pParent = pNode->pParent; + pNode->pParent = 0; + rc = deleteCell(pRtree, pParent, iCell, iHeight+1); + } + rc2 = nodeRelease(pRtree, pParent); + if( rc==SQLITE_OK ){ + rc = rc2; + } + if( rc!=SQLITE_OK ){ return rc; } /* Remove the xxx_node entry. */ sqlite3_bind_int64(pRtree->pDeleteNode, 1, pNode->iNode); @@ -117381,12 +119099,13 @@ pRtree->pDeleted = pNode; return SQLITE_OK; } -static void fixBoundingBox(Rtree *pRtree, RtreeNode *pNode){ +static int fixBoundingBox(Rtree *pRtree, RtreeNode *pNode){ RtreeNode *pParent = pNode->pParent; + int rc = SQLITE_OK; if( pParent ){ int ii; int nCell = NCELL(pNode); RtreeCell box; /* Bounding box for pNode */ nodeGetCell(pRtree, pNode, 0, &box); @@ -117394,21 +119113,25 @@ RtreeCell cell; nodeGetCell(pRtree, pNode, ii, &cell); cellUnion(pRtree, &box, &cell); } box.iRowid = pNode->iNode; - ii = nodeParentIndex(pRtree, pNode); - nodeOverwriteCell(pRtree, pParent, &box, ii); - fixBoundingBox(pRtree, pParent); + rc = nodeParentIndex(pRtree, pNode, &ii); + if( rc==SQLITE_OK ){ + nodeOverwriteCell(pRtree, pParent, &box, ii); + rc = fixBoundingBox(pRtree, pParent); + } } + return rc; } /* ** Delete the cell at index iCell of node pNode. After removing the ** cell, adjust the r-tree data structure if required. */ static int deleteCell(Rtree *pRtree, RtreeNode *pNode, int iCell, int iHeight){ + RtreeNode *pParent; int rc; if( SQLITE_OK!=(rc = fixLeafParent(pRtree, pNode)) ){ return rc; } @@ -117421,18 +119144,17 @@ /* If the node is not the tree root and now has less than the minimum ** number of cells, remove it from the tree. Otherwise, update the ** cell in the parent node so that it tightly contains the updated ** node. */ - if( pNode->iNode!=1 ){ - RtreeNode *pParent = pNode->pParent; - if( (pParent->iNode!=1 || NCELL(pParent)!=1) - && (NCELL(pNode)<RTREE_MINCELLS(pRtree)) - ){ + pParent = pNode->pParent; + assert( pParent || pNode->iNode==1 ); + if( pParent ){ + if( NCELL(pNode)<RTREE_MINCELLS(pRtree) ){ rc = removeNode(pRtree, pNode, iHeight); }else{ - fixBoundingBox(pRtree, pNode); + rc = fixBoundingBox(pRtree, pNode); } } return rc; } @@ -117511,11 +119233,11 @@ rc = parentWrite(pRtree, p->iRowid, pNode->iNode); } } } if( rc==SQLITE_OK ){ - fixBoundingBox(pRtree, pNode); + rc = fixBoundingBox(pRtree, pNode); } for(; rc==SQLITE_OK && ii<nCell; ii++){ /* Find a node to store this cell in. pNode->iNode currently contains ** the height of the sub-tree headed by the cell. */ @@ -117565,15 +119287,17 @@ } #else rc = SplitNode(pRtree, pNode, pCell, iHeight); #endif }else{ - AdjustTree(pRtree, pNode, pCell); - if( iHeight==0 ){ - rc = rowidWrite(pRtree, pCell->iRowid, pNode->iNode); - }else{ - rc = parentWrite(pRtree, pCell->iRowid, pNode->iNode); + rc = AdjustTree(pRtree, pNode, pCell); + if( rc==SQLITE_OK ){ + if( iHeight==0 ){ + rc = rowidWrite(pRtree, pCell->iRowid, pNode->iNode); + }else{ + rc = parentWrite(pRtree, pCell->iRowid, pNode->iNode); + } } } return rc; } @@ -117639,11 +119363,10 @@ int rc = SQLITE_OK; rtreeReference(pRtree); assert(nData>=1); - assert(hashIsEmpty(pRtree)); /* If azData[0] is not an SQL NULL value, it is the rowid of a ** record to delete from the r-tree table. The following block does ** just that. */ @@ -117665,12 +119388,14 @@ } /* Delete the cell in question from the leaf node. */ if( rc==SQLITE_OK ){ int rc2; - iCell = nodeRowidIndex(pRtree, pLeaf, iDelete); - rc = deleteCell(pRtree, pLeaf, iCell, 0); + rc = nodeRowidIndex(pRtree, pLeaf, iDelete, &iCell); + if( rc==SQLITE_OK ){ + rc = deleteCell(pRtree, pLeaf, iCell, 0); + } rc2 = nodeRelease(pRtree, pLeaf); if( rc==SQLITE_OK ){ rc = rc2; } } @@ -117688,23 +119413,24 @@ ** ** This is equivalent to copying the contents of the child into ** the root node (the operation that Gutman's paper says to perform ** in this scenario). */ - if( rc==SQLITE_OK && pRtree->iDepth>0 ){ - if( rc==SQLITE_OK && NCELL(pRoot)==1 ){ - RtreeNode *pChild; - i64 iChild = nodeGetRowid(pRtree, pRoot, 0); - rc = nodeAcquire(pRtree, iChild, pRoot, &pChild); - if( rc==SQLITE_OK ){ - rc = removeNode(pRtree, pChild, pRtree->iDepth-1); - } - if( rc==SQLITE_OK ){ - pRtree->iDepth--; - writeInt16(pRoot->zData, pRtree->iDepth); - pRoot->isDirty = 1; - } + if( rc==SQLITE_OK && pRtree->iDepth>0 && NCELL(pRoot)==1 ){ + int rc2; + RtreeNode *pChild; + i64 iChild = nodeGetRowid(pRtree, pRoot, 0); + rc = nodeAcquire(pRtree, iChild, pRoot, &pChild); + if( rc==SQLITE_OK ){ + rc = removeNode(pRtree, pChild, pRtree->iDepth-1); + } + rc2 = nodeRelease(pRtree, pChild); + if( rc==SQLITE_OK ) rc = rc2; + if( rc==SQLITE_OK ){ + pRtree->iDepth--; + writeInt16(pRoot->zData, pRtree->iDepth); + pRoot->isDirty = 1; } } /* Re-insert the contents of any underfull nodes removed from the tree. */ for(pLeaf=pRtree->pDeleted; pLeaf; pLeaf=pRtree->pDeleted){ @@ -117990,11 +119716,11 @@ ){ int rc = SQLITE_OK; Rtree *pRtree; int nDb; /* Length of string argv[1] */ int nName; /* Length of string argv[2] */ - int eCoordType = (int)pAux; + int eCoordType = (pAux ? RTREE_COORD_INT32 : RTREE_COORD_REAL32); const char *aErrMsg[] = { 0, /* 0 */ "Wrong number of columns for an rtree table", /* 1 */ "Too few columns for an rtree table", /* 2 */ @@ -118136,16 +119862,14 @@ ** Register the r-tree module with database handle db. This creates the ** virtual table module "rtree" and the debugging/analysis scalar ** function "rtreenode". */ SQLITE_PRIVATE int sqlite3RtreeInit(sqlite3 *db){ - int rc = SQLITE_OK; + const int utf8 = SQLITE_UTF8; + int rc; - if( rc==SQLITE_OK ){ - int utf8 = SQLITE_UTF8; - rc = sqlite3_create_function(db, "rtreenode", 2, utf8, 0, rtreenode, 0, 0); - } + rc = sqlite3_create_function(db, "rtreenode", 2, utf8, 0, rtreenode, 0, 0); if( rc==SQLITE_OK ){ int utf8 = SQLITE_UTF8; rc = sqlite3_create_function(db, "rtreedepth", 1, utf8, 0,rtreedepth, 0, 0); } if( rc==SQLITE_OK ){ @@ -118157,10 +119881,74 @@ rc = sqlite3_create_module_v2(db, "rtree_i32", &rtreeModule, c, 0); } return rc; } + +/* +** A version of sqlite3_free() that can be used as a callback. This is used +** in two places - as the destructor for the blob value returned by the +** invocation of a geometry function, and as the destructor for the geometry +** functions themselves. +*/ +static void doSqlite3Free(void *p){ + sqlite3_free(p); +} + +/* +** Each call to sqlite3_rtree_geometry_callback() creates an ordinary SQLite +** scalar user function. This C function is the callback used for all such +** registered SQL functions. +** +** The scalar user functions return a blob that is interpreted by r-tree +** table MATCH operators. +*/ +static void geomCallback(sqlite3_context *ctx, int nArg, sqlite3_value **aArg){ + RtreeGeomCallback *pGeomCtx = (RtreeGeomCallback *)sqlite3_user_data(ctx); + RtreeMatchArg *pBlob; + int nBlob; + + nBlob = sizeof(RtreeMatchArg) + (nArg-1)*sizeof(double); + pBlob = (RtreeMatchArg *)sqlite3_malloc(nBlob); + if( !pBlob ){ + sqlite3_result_error_nomem(ctx); + }else{ + int i; + pBlob->magic = RTREE_GEOMETRY_MAGIC; + pBlob->xGeom = pGeomCtx->xGeom; + pBlob->pContext = pGeomCtx->pContext; + pBlob->nParam = nArg; + for(i=0; i<nArg; i++){ + pBlob->aParam[i] = sqlite3_value_double(aArg[i]); + } + sqlite3_result_blob(ctx, pBlob, nBlob, doSqlite3Free); + } +} + +/* +** Register a new geometry function for use with the r-tree MATCH operator. +*/ +SQLITE_API int sqlite3_rtree_geometry_callback( + sqlite3 *db, + const char *zGeom, + int (*xGeom)(sqlite3_rtree_geometry *, int, double *, int *), + void *pContext +){ + RtreeGeomCallback *pGeomCtx; /* Context object for new user-function */ + + /* Allocate and populate the context object. */ + pGeomCtx = (RtreeGeomCallback *)sqlite3_malloc(sizeof(RtreeGeomCallback)); + if( !pGeomCtx ) return SQLITE_NOMEM; + pGeomCtx->xGeom = xGeom; + pGeomCtx->pContext = pContext; + + /* Create the new user-function. Register a destructor function to delete + ** the context object when it is no longer required. */ + return sqlite3_create_function_v2(db, zGeom, -1, SQLITE_ANY, + (void *)pGeomCtx, geomCallback, 0, 0, doSqlite3Free + ); +} #if !SQLITE_CORE SQLITE_API int sqlite3_extension_init( sqlite3 *db, char **pzErrMsg, Index: src/sqlite3.h ================================================================== --- src/sqlite3.h +++ src/sqlite3.h @@ -95,23 +95,23 @@ ** be held constant and Z will be incremented or else Y will be incremented ** and Z will be reset to zero. ** ** Since version 3.6.18, SQLite source code has been stored in the ** <a href="http://www.fossil-scm.org/">Fossil configuration management -** system</a>. ^The SQLITE_SOURCE_ID macro evalutes to +** system</a>. ^The SQLITE_SOURCE_ID macro evaluates to ** a string which identifies a particular check-in of SQLite ** within its configuration management system. ^The SQLITE_SOURCE_ID ** string contains the date and time of the check-in (UTC) and an SHA1 ** hash of the entire source tree. ** ** See also: [sqlite3_libversion()], ** [sqlite3_libversion_number()], [sqlite3_sourceid()], ** [sqlite_version()] and [sqlite_source_id()]. */ -#define SQLITE_VERSION "3.7.1" -#define SQLITE_VERSION_NUMBER 3007001 -#define SQLITE_SOURCE_ID "2010-08-05 03:21:40 fbe70e1106bcc5086ceb9d8f39cc39baf3643092" +#define SQLITE_VERSION "3.7.3" +#define SQLITE_VERSION_NUMBER 3007003 +#define SQLITE_SOURCE_ID "2010-10-07 13:29:13 e55ada89246d4cc5f476891c70572dc7c1c3643e" /* ** CAPI3REF: Run-Time Library Version Numbers ** KEYWORDS: sqlite3_version, sqlite3_sourceid ** @@ -152,19 +152,19 @@ ** ^The sqlite3_compileoption_used() function returns 0 or 1 ** indicating whether the specified option was defined at ** compile time. ^The SQLITE_ prefix may be omitted from the ** option name passed to sqlite3_compileoption_used(). ** -** ^The sqlite3_compileoption_get() function allows interating +** ^The sqlite3_compileoption_get() function allows iterating ** over the list of options that were defined at compile time by ** returning the N-th compile time option string. ^If N is out of range, ** sqlite3_compileoption_get() returns a NULL pointer. ^The SQLITE_ ** prefix is omitted from any strings returned by ** sqlite3_compileoption_get(). ** ** ^Support for the diagnostic functions sqlite3_compileoption_used() -** and sqlite3_compileoption_get() may be omitted by specifing the +** and sqlite3_compileoption_get() may be omitted by specifying the ** [SQLITE_OMIT_COMPILEOPTION_DIAGS] option at compile time. ** ** See also: SQL functions [sqlite_compileoption_used()] and ** [sqlite_compileoption_get()] and the [compile_options pragma]. */ @@ -266,11 +266,11 @@ /* ** CAPI3REF: Closing A Database Connection ** ** ^The sqlite3_close() routine is the destructor for the [sqlite3] object. ** ^Calls to sqlite3_close() return SQLITE_OK if the [sqlite3] object is -** successfullly destroyed and all associated resources are deallocated. +** successfully destroyed and all associated resources are deallocated. ** ** Applications must [sqlite3_finalize | finalize] all [prepared statements] ** and [sqlite3_blob_close | close] all [BLOB handles] associated with ** the [sqlite3] object prior to attempting to close the object. ^If ** sqlite3_close() is called on a [database connection] that still has @@ -693,16 +693,25 @@ ** layer a hint of how large the database file will grow to be during the ** current transaction. This hint is not guaranteed to be accurate but it ** is often close. The underlying VFS might choose to preallocate database ** file space based on this hint in order to help writes to the database ** file run faster. +** +** The [SQLITE_FCNTL_CHUNK_SIZE] opcode is used to request that the VFS +** extends and truncates the database file in chunks of a size specified +** by the user. The fourth argument to [sqlite3_file_control()] should +** point to an integer (type int) containing the new chunk-size to use +** for the nominated database. Allocating database file space in large +** chunks (say 1MB at a time), may reduce file-system fragmentation and +** improve performance on some systems. */ #define SQLITE_FCNTL_LOCKSTATE 1 #define SQLITE_GET_LOCKPROXYFILE 2 #define SQLITE_SET_LOCKPROXYFILE 3 #define SQLITE_LAST_ERRNO 4 #define SQLITE_FCNTL_SIZE_HINT 5 +#define SQLITE_FCNTL_CHUNK_SIZE 6 /* ** CAPI3REF: Mutex Handle ** ** The mutex module within SQLite defines [sqlite3_mutex] to be an @@ -746,19 +755,23 @@ ** object once the object has been registered. ** ** The zName field holds the name of the VFS module. The name must ** be unique across all VFS modules. ** -** SQLite will guarantee that the zFilename parameter to xOpen +** ^SQLite guarantees that the zFilename parameter to xOpen ** is either a NULL pointer or string obtained -** from xFullPathname(). SQLite further guarantees that +** from xFullPathname() with an optional suffix added. +** ^If a suffix is added to the zFilename parameter, it will +** consist of a single "-" character followed by no more than +** 10 alphanumeric and/or "-" characters. +** ^SQLite further guarantees that ** the string will be valid and unchanged until xClose() is ** called. Because of the previous sentence, ** the [sqlite3_file] can safely store a pointer to the ** filename if it needs to remember the filename for some reason. -** If the zFilename parameter is xOpen is a NULL pointer then xOpen -** must invent its own temporary name for the file. Whenever the +** If the zFilename parameter to xOpen is a NULL pointer then xOpen +** must invent its own temporary name for the file. ^Whenever the ** xFilename parameter is NULL it will also be the case that the ** flags parameter will include [SQLITE_OPEN_DELETEONCLOSE]. ** ** The flags argument to xOpen() includes all bits set in ** the flags argument to [sqlite3_open_v2()]. Or if [sqlite3_open()] @@ -765,11 +778,11 @@ ** or [sqlite3_open16()] is used, then flags includes at least ** [SQLITE_OPEN_READWRITE] | [SQLITE_OPEN_CREATE]. ** If xOpen() opens a file read-only then it sets *pOutFlags to ** include [SQLITE_OPEN_READONLY]. Other bits in *pOutFlags may be set. ** -** SQLite will also add one of the following flags to the xOpen() +** ^(SQLite will also add one of the following flags to the xOpen() ** call, depending on the object being opened: ** ** <ul> ** <li> [SQLITE_OPEN_MAIN_DB] ** <li> [SQLITE_OPEN_MAIN_JOURNAL] @@ -776,11 +789,12 @@ ** <li> [SQLITE_OPEN_TEMP_DB] ** <li> [SQLITE_OPEN_TEMP_JOURNAL] ** <li> [SQLITE_OPEN_TRANSIENT_DB] ** <li> [SQLITE_OPEN_SUBJOURNAL] ** <li> [SQLITE_OPEN_MASTER_JOURNAL] -** </ul> +** <li> [SQLITE_OPEN_WAL] +** </ul>)^ ** ** The file I/O implementation can use the object type flags to ** change the way it deals with files. For example, an application ** that does not care about crash recovery or rollback might make ** the open of a journal file a no-op. Writes to this journal would @@ -795,39 +809,40 @@ ** <li> [SQLITE_OPEN_DELETEONCLOSE] ** <li> [SQLITE_OPEN_EXCLUSIVE] ** </ul> ** ** The [SQLITE_OPEN_DELETEONCLOSE] flag means the file should be -** deleted when it is closed. The [SQLITE_OPEN_DELETEONCLOSE] -** will be set for TEMP databases, journals and for subjournals. +** deleted when it is closed. ^The [SQLITE_OPEN_DELETEONCLOSE] +** will be set for TEMP databases and their journals, transient +** databases, and subjournals. ** -** The [SQLITE_OPEN_EXCLUSIVE] flag is always used in conjunction +** ^The [SQLITE_OPEN_EXCLUSIVE] flag is always used in conjunction ** with the [SQLITE_OPEN_CREATE] flag, which are both directly ** analogous to the O_EXCL and O_CREAT flags of the POSIX open() ** API. The SQLITE_OPEN_EXCLUSIVE flag, when paired with the ** SQLITE_OPEN_CREATE, is used to indicate that file should always ** be created, and that it is an error if it already exists. ** It is <i>not</i> used to indicate the file should be opened ** for exclusive access. ** -** At least szOsFile bytes of memory are allocated by SQLite +** ^At least szOsFile bytes of memory are allocated by SQLite ** to hold the [sqlite3_file] structure passed as the third ** argument to xOpen. The xOpen method does not have to ** allocate the structure; it should just fill it in. Note that ** the xOpen method must set the sqlite3_file.pMethods to either ** a valid [sqlite3_io_methods] object or to NULL. xOpen must do ** this even if the open fails. SQLite expects that the sqlite3_file.pMethods ** element will be valid after xOpen returns regardless of the success ** or failure of the xOpen call. ** -** The flags argument to xAccess() may be [SQLITE_ACCESS_EXISTS] +** ^The flags argument to xAccess() may be [SQLITE_ACCESS_EXISTS] ** to test for the existence of a file, or [SQLITE_ACCESS_READWRITE] to ** test whether a file is readable and writable, or [SQLITE_ACCESS_READ] ** to test whether a file is at least readable. The file can be a ** directory. ** -** SQLite will always allocate at least mxPathname+1 bytes for the +** ^SQLite will always allocate at least mxPathname+1 bytes for the ** output buffer xFullPathname. The exact size of the output buffer ** is also passed as a parameter to both methods. If the output buffer ** is not large enough, [SQLITE_CANTOPEN] should be returned. Since this is ** handled as a fatal error by SQLite, vfs implementations should endeavor ** to prevent this by setting mxPathname to a sufficiently large value. @@ -837,14 +852,14 @@ ** included in the VFS structure for completeness. ** The xRandomness() function attempts to return nBytes bytes ** of good-quality randomness into zOut. The return value is ** the actual number of bytes of randomness obtained. ** The xSleep() method causes the calling thread to sleep for at -** least the number of microseconds given. The xCurrentTime() +** least the number of microseconds given. ^The xCurrentTime() ** method returns a Julian Day Number for the current date and time as ** a floating point value. -** The xCurrentTimeInt64() method returns, as an integer, the Julian +** ^The xCurrentTimeInt64() method returns, as an integer, the Julian ** Day Number multipled by 86400000 (the number of milliseconds in ** a 24-hour day). ** ^SQLite will use the xCurrentTimeInt64() method to get the current ** date and time if that method is available (if iVersion is 2 or ** greater and the function pointer is not NULL) and will fall back @@ -1237,11 +1252,11 @@ ** statistics. ^(When memory allocation statistics are disabled, the ** following SQLite interfaces become non-operational: ** <ul> ** <li> [sqlite3_memory_used()] ** <li> [sqlite3_memory_highwater()] -** <li> [sqlite3_soft_heap_limit()] +** <li> [sqlite3_soft_heap_limit64()] ** <li> [sqlite3_status()] ** </ul>)^ ** ^Memory allocation statistics are enabled by default unless SQLite is ** compiled with [SQLITE_DEFAULT_MEMSTATUS]=0 in which case memory ** allocation statistics are disabled by default. @@ -1251,19 +1266,18 @@ ** <dd> ^This option specifies a static memory buffer that SQLite can use for ** scratch memory. There are three arguments: A pointer an 8-byte ** aligned memory buffer from which the scrach allocations will be ** drawn, the size of each scratch allocation (sz), ** and the maximum number of scratch allocations (N). The sz -** argument must be a multiple of 16. The sz parameter should be a few bytes -** larger than the actual scratch space required due to internal overhead. +** argument must be a multiple of 16. ** The first argument must be a pointer to an 8-byte aligned buffer ** of at least sz*N bytes of memory. -** ^SQLite will use no more than one scratch buffer per thread. So -** N should be set to the expected maximum number of threads. ^SQLite will -** never require a scratch buffer that is more than 6 times the database -** page size. ^If SQLite needs needs additional scratch memory beyond -** what is provided by this configuration option, then +** ^SQLite will use no more than two scratch buffers per thread. So +** N should be set to twice the expected maximum number of threads. +** ^SQLite will never require a scratch buffer that is more than 6 +** times the database page size. ^If SQLite needs needs additional +** scratch memory beyond what is provided by this configuration option, then ** [sqlite3_malloc()] will be used to obtain the memory needed.</dd> ** ** <dt>SQLITE_CONFIG_PAGECACHE</dt> ** <dd> ^This option specifies a static memory buffer that SQLite can use for ** the database page cache with the default page cache implemenation. @@ -1279,12 +1293,11 @@ ** argument should point to an allocation of at least sz*N bytes of memory. ** ^SQLite will use the memory provided by the first argument to satisfy its ** memory needs for the first N pages that it adds to cache. ^If additional ** page cache memory is needed beyond what is provided by this option, then ** SQLite goes to [sqlite3_malloc()] for the additional storage space. -** ^The implementation might use one or more of the N buffers to hold -** memory accounting information. The pointer in the first argument must +** The pointer in the first argument must ** be aligned to an 8-byte boundary or subsequent behavior of SQLite ** will be undefined.</dd> ** ** <dt>SQLITE_CONFIG_HEAP</dt> ** <dd> ^This option specifies a static memory buffer that SQLite will use @@ -1409,12 +1422,18 @@ ** size of each lookaside buffer slot. ^The third argument is the number of ** slots. The size of the buffer in the first argument must be greater than ** or equal to the product of the second and third arguments. The buffer ** must be aligned to an 8-byte boundary. ^If the second argument to ** SQLITE_DBCONFIG_LOOKASIDE is not a multiple of 8, it is internally -** rounded down to the next smaller -** multiple of 8. See also: [SQLITE_CONFIG_LOOKASIDE]</dd> +** rounded down to the next smaller multiple of 8. ^(The lookaside memory +** configuration for a database connection can only be changed when that +** connection is not currently using lookaside memory, or in other words +** when the "current value" returned by +** [sqlite3_db_status](D,[SQLITE_CONFIG_LOOKASIDE],...) is zero. +** Any attempt to change the lookaside memory configuration when lookaside +** memory is in use leaves the configuration unchanged and returns +** [SQLITE_BUSY].)^</dd> ** ** </dl> */ #define SQLITE_DBCONFIG_LOOKASIDE 1001 /* void* int int */ @@ -1714,10 +1733,13 @@ */ SQLITE_API int sqlite3_busy_timeout(sqlite3*, int ms); /* ** CAPI3REF: Convenience Routines For Running Queries +** +** This is a legacy interface that is preserved for backwards compatibility. +** Use of this interface is not recommended. ** ** Definition: A <b>result table</b> is memory data structure created by the ** [sqlite3_get_table()] interface. A result table records the ** complete query results from one or more queries. ** @@ -1735,11 +1757,11 @@ ** ** A result table might consist of one or more memory allocations. ** It is not safe to pass a result table directly to [sqlite3_free()]. ** A result table should be deallocated using [sqlite3_free_table()]. ** -** As an example of the result table format, suppose a query result +** ^(As an example of the result table format, suppose a query result ** is as follows: ** ** <blockquote><pre> ** Name | Age ** ----------------------- @@ -1759,31 +1781,31 @@ ** azResult[3] = "43"; ** azResult[4] = "Bob"; ** azResult[5] = "28"; ** azResult[6] = "Cindy"; ** azResult[7] = "21"; -** </pre></blockquote> +** </pre></blockquote>)^ ** ** ^The sqlite3_get_table() function evaluates one or more ** semicolon-separated SQL statements in the zero-terminated UTF-8 ** string of its 2nd parameter and returns a result table to the ** pointer given in its 3rd parameter. ** ** After the application has finished with the result from sqlite3_get_table(), -** it should pass the result table pointer to sqlite3_free_table() in order to +** it must pass the result table pointer to sqlite3_free_table() in order to ** release the memory that was malloced. Because of the way the ** [sqlite3_malloc()] happens within sqlite3_get_table(), the calling ** function must not try to call [sqlite3_free()] directly. Only ** [sqlite3_free_table()] is able to release the memory properly and safely. ** -** ^(The sqlite3_get_table() interface is implemented as a wrapper around +** The sqlite3_get_table() interface is implemented as a wrapper around ** [sqlite3_exec()]. The sqlite3_get_table() routine does not have access ** to any internal data structures of SQLite. It uses only the public ** interface defined here. As a consequence, errors that occur in the ** wrapper layer outside of the internal [sqlite3_exec()] call are not ** reflected in subsequent calls to [sqlite3_errcode()] or -** [sqlite3_errmsg()].)^ +** [sqlite3_errmsg()]. */ SQLITE_API int sqlite3_get_table( sqlite3 *db, /* An open database */ const char *zSql, /* SQL to be evaluated */ char ***pazResult, /* Results of the query */ @@ -1931,11 +1953,13 @@ ** by sqlite3_realloc() and the prior allocation is freed. ** ^If sqlite3_realloc() returns NULL, then the prior allocation ** is not freed. ** ** ^The memory returned by sqlite3_malloc() and sqlite3_realloc() -** is always aligned to at least an 8 byte boundary. +** is always aligned to at least an 8 byte boundary, or to a +** 4 byte boundary if the [SQLITE_4_BYTE_ALIGNED_MALLOC] compile-time +** option is used. ** ** In SQLite version 3.5.0 and 3.5.1, it was possible to define ** the SQLITE_OMIT_MEMORY_ALLOCATION which would cause the built-in ** implementation of these routines to be omitted. That capability ** is no longer provided. Only built-in memory allocators can be used. @@ -2189,21 +2213,32 @@ void(*xProfile)(void*,const char*,sqlite3_uint64), void*); /* ** CAPI3REF: Query Progress Callbacks ** -** ^This routine configures a callback function - the -** progress callback - that is invoked periodically during long -** running calls to [sqlite3_exec()], [sqlite3_step()] and -** [sqlite3_get_table()]. An example use for this +** ^The sqlite3_progress_handler(D,N,X,P) interface causes the callback +** function X to be invoked periodically during long running calls to +** [sqlite3_exec()], [sqlite3_step()] and [sqlite3_get_table()] for +** database connection D. An example use for this ** interface is to keep a GUI updated during a large query. +** +** ^The parameter P is passed through as the only parameter to the +** callback function X. ^The parameter N is the number of +** [virtual machine instructions] that are evaluated between successive +** invocations of the callback X. +** +** ^Only a single progress handler may be defined at one time per +** [database connection]; setting a new progress handler cancels the +** old one. ^Setting parameter X to NULL disables the progress handler. +** ^The progress handler is also disabled by setting N to a value less +** than 1. ** ** ^If the progress callback returns non-zero, the operation is ** interrupted. This feature can be used to implement a ** "Cancel" button on a GUI progress dialog box. ** -** The progress handler must not do anything that will modify +** The progress handler callback must not do anything that will modify ** the database connection that invoked the progress handler. ** Note that [sqlite3_prepare_v2()] and [sqlite3_step()] both modify their ** database connections for the meaning of "modify" in this paragraph. ** */ @@ -2258,11 +2293,11 @@ ** </dl> ** ** If the 3rd parameter to sqlite3_open_v2() is not one of the ** combinations shown above or one of the combinations shown above combined ** with the [SQLITE_OPEN_NOMUTEX], [SQLITE_OPEN_FULLMUTEX], -** [SQLITE_OPEN_SHAREDCACHE] and/or [SQLITE_OPEN_SHAREDCACHE] flags, +** [SQLITE_OPEN_SHAREDCACHE] and/or [SQLITE_OPEN_PRIVATECACHE] flags, ** then the behavior is undefined. ** ** ^If the [SQLITE_OPEN_NOMUTEX] flag is set, then the database connection ** opens in the multi-thread [threading mode] as long as the single-thread ** mode has not been set at compile-time or start-time. ^If the @@ -2383,21 +2418,26 @@ ** ^(This interface allows the size of various constructs to be limited ** on a connection by connection basis. The first parameter is the ** [database connection] whose limit is to be set or queried. The ** second parameter is one of the [limit categories] that define a ** class of constructs to be size limited. The third parameter is the -** new limit for that construct. The function returns the old limit.)^ +** new limit for that construct.)^ ** ** ^If the new limit is a negative number, the limit is unchanged. -** ^(For the limit category of SQLITE_LIMIT_XYZ there is a +** ^(For each limit category SQLITE_LIMIT_<i>NAME</i> there is a ** [limits | hard upper bound] -** set by a compile-time C preprocessor macro named -** [limits | SQLITE_MAX_XYZ]. +** set at compile-time by a C preprocessor macro called +** [limits | SQLITE_MAX_<i>NAME</i>]. ** (The "_LIMIT_" in the name is changed to "_MAX_".))^ ** ^Attempts to increase a limit above its hard upper bound are ** silently truncated to the hard upper bound. ** +** ^Regardless of whether or not the limit was changed, the +** [sqlite3_limit()] interface returns the prior value of the limit. +** ^Hence, to find the current value of a limit without changing it, +** simply invoke this interface with the third parameter set to -1. +** ** Run-time limits are intended for use in applications that manage ** both their own internal database and also databases that are controlled ** by untrusted external sources. An example application might be a ** web browser that has its own databases for storing history and ** separate databases controlled by JavaScript applications downloaded @@ -2422,11 +2462,11 @@ ** The synopsis of the meanings of the various limits is shown below. ** Additional information is available at [limits | Limits in SQLite]. ** ** <dl> ** ^(<dt>SQLITE_LIMIT_LENGTH</dt> -** <dd>The maximum size of any string or BLOB or table row.<dd>)^ +** <dd>The maximum size of any string or BLOB or table row, in bytes.<dd>)^ ** ** ^(<dt>SQLITE_LIMIT_SQL_LENGTH</dt> ** <dd>The maximum length of an SQL statement, in bytes.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_COLUMN</dt> @@ -2440,11 +2480,13 @@ ** ^(<dt>SQLITE_LIMIT_COMPOUND_SELECT</dt> ** <dd>The maximum number of terms in a compound SELECT statement.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_VDBE_OP</dt> ** <dd>The maximum number of instructions in a virtual machine program -** used to implement an SQL statement.</dd>)^ +** used to implement an SQL statement. This limit is not currently +** enforced, though that might be added in some future release of +** SQLite.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_FUNCTION_ARG</dt> ** <dd>The maximum number of arguments on a function.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_ATTACHED</dt> @@ -2453,12 +2495,11 @@ ** ^(<dt>SQLITE_LIMIT_LIKE_PATTERN_LENGTH</dt> ** <dd>The maximum length of the pattern argument to the [LIKE] or ** [GLOB] operators.</dd>)^ ** ** ^(<dt>SQLITE_LIMIT_VARIABLE_NUMBER</dt> -** <dd>The maximum number of variables in an SQL statement that can -** be bound.</dd>)^ +** <dd>The maximum index number of any [parameter] in an SQL statement.)^ ** ** ^(<dt>SQLITE_LIMIT_TRIGGER_DEPTH</dt> ** <dd>The maximum depth of recursion for triggers.</dd>)^ ** </dl> */ @@ -2526,16 +2567,11 @@ ** ** <ol> ** <li> ** ^If the database schema changes, instead of returning [SQLITE_SCHEMA] as it ** always used to do, [sqlite3_step()] will automatically recompile the SQL -** statement and try to run it again. ^If the schema has changed in -** a way that makes the statement no longer valid, [sqlite3_step()] will still -** return [SQLITE_SCHEMA]. But unlike the legacy behavior, [SQLITE_SCHEMA] is -** now a fatal error. Calling [sqlite3_prepare_v2()] again will not make the -** error go away. Note: use [sqlite3_errmsg()] to find the text -** of the parsing error that results in an [SQLITE_SCHEMA] return. +** statement and try to run it again. ** </li> ** ** <li> ** ^When an error occurs, [sqlite3_step()] will return one of the detailed ** [error codes] or [extended error codes]. ^The legacy behavior was that @@ -2544,15 +2580,20 @@ ** in order to find the underlying cause of the problem. With the "v2" prepare ** interfaces, the underlying reason for the error is returned immediately. ** </li> ** ** <li> -** ^If the value of a [parameter | host parameter] in the WHERE clause might -** change the query plan for a statement, then the statement may be -** automatically recompiled (as if there had been a schema change) on the first -** [sqlite3_step()] call following any change to the -** [sqlite3_bind_text | bindings] of the [parameter]. +** ^If the specific value bound to [parameter | host parameter] in the +** WHERE clause might influence the choice of query plan for a statement, +** then the statement will be automatically recompiled, as if there had been +** a schema change, on the first [sqlite3_step()] call following any change +** to the [sqlite3_bind_text | bindings] of that [parameter]. +** ^The specific value of WHERE-clause [parameter] might influence the +** choice of query plan if the parameter is the left-hand side of a [LIKE] +** or [GLOB] operator or if the parameter is compared to an indexed column +** and the [SQLITE_ENABLE_STAT2] compile-time option is enabled. +** the ** </li> ** </ol> */ SQLITE_API int sqlite3_prepare( sqlite3 *db, /* Database handle */ @@ -2615,11 +2656,11 @@ ** or if SQLite is run in one of reduced mutex modes ** [SQLITE_CONFIG_SINGLETHREAD] or [SQLITE_CONFIG_MULTITHREAD] ** then there is no distinction between protected and unprotected ** sqlite3_value objects and they can be used interchangeably. However, ** for maximum code portability it is recommended that applications -** still make the distinction between between protected and unprotected +** still make the distinction between protected and unprotected ** sqlite3_value objects even when not strictly required. ** ** ^The sqlite3_value objects that are passed as parameters into the ** implementation of [application-defined SQL functions] are protected. ** ^The sqlite3_value object returned by @@ -2661,11 +2702,11 @@ ** <li> @VVV ** <li> $VVV ** </ul> ** ** In the templates above, NNN represents an integer literal, -** and VVV represents an alphanumeric identifer.)^ ^The values of these +** and VVV represents an alphanumeric identifier.)^ ^The values of these ** parameters (also called "host parameter names" or "SQL parameters") ** can be set using the sqlite3_bind_*() routines defined here. ** ** ^The first argument to the sqlite3_bind_*() routines is always ** a pointer to the [sqlite3_stmt] object returned from @@ -2810,10 +2851,12 @@ ** CAPI3REF: Number Of Columns In A Result Set ** ** ^Return the number of columns in the result set returned by the ** [prepared statement]. ^This routine returns 0 if pStmt is an SQL ** statement that does not return data (for example an [UPDATE]). +** +** See also: [sqlite3_data_count()] */ SQLITE_API int sqlite3_column_count(sqlite3_stmt *pStmt); /* ** CAPI3REF: Column Names In A Result Set @@ -3000,12 +3043,18 @@ SQLITE_API int sqlite3_step(sqlite3_stmt*); /* ** CAPI3REF: Number of columns in a result set ** -** ^The sqlite3_data_count(P) the number of columns in the -** of the result set of [prepared statement] P. +** ^The sqlite3_data_count(P) interface returns the number of columns in the +** current row of the result set of [prepared statement] P. +** ^If prepared statement P does not have results ready to return +** (via calls to the [sqlite3_column_int | sqlite3_column_*()] of +** interfaces) then sqlite3_data_count(P) returns 0. +** ^The sqlite3_data_count(P) routine also returns 0 if P is a NULL pointer. +** +** See also: [sqlite3_column_count()] */ SQLITE_API int sqlite3_data_count(sqlite3_stmt *pStmt); /* ** CAPI3REF: Fundamental Datatypes @@ -3081,22 +3130,30 @@ ** ^If the result is a UTF-16 string, then sqlite3_column_bytes() converts ** the string to UTF-8 and then returns the number of bytes. ** ^If the result is a numeric value then sqlite3_column_bytes() uses ** [sqlite3_snprintf()] to convert that value to a UTF-8 string and returns ** the number of bytes in that string. -** ^The value returned does not include the zero terminator at the end -** of the string. ^For clarity: the value returned is the number of +** ^If the result is NULL, then sqlite3_column_bytes() returns zero. +** +** ^If the result is a BLOB or UTF-16 string then the sqlite3_column_bytes16() +** routine returns the number of bytes in that BLOB or string. +** ^If the result is a UTF-8 string, then sqlite3_column_bytes16() converts +** the string to UTF-16 and then returns the number of bytes. +** ^If the result is a numeric value then sqlite3_column_bytes16() uses +** [sqlite3_snprintf()] to convert that value to a UTF-16 string and returns +** the number of bytes in that string. +** ^If the result is NULL, then sqlite3_column_bytes16() returns zero. +** +** ^The values returned by [sqlite3_column_bytes()] and +** [sqlite3_column_bytes16()] do not include the zero terminators at the end +** of the string. ^For clarity: the values returned by +** [sqlite3_column_bytes()] and [sqlite3_column_bytes16()] are the number of ** bytes in the string, not the number of characters. ** ** ^Strings returned by sqlite3_column_text() and sqlite3_column_text16(), ** even empty strings, are always zero terminated. ^The return -** value from sqlite3_column_blob() for a zero-length BLOB is an arbitrary -** pointer, possibly even a NULL pointer. -** -** ^The sqlite3_column_bytes16() routine is similar to sqlite3_column_bytes() -** but leaves the result in UTF-16 in native byte order instead of UTF-8. -** ^The zero terminator is not included in this count. +** value from sqlite3_column_blob() for a zero-length BLOB is a NULL pointer. ** ** ^The object returned by [sqlite3_column_value()] is an ** [unprotected sqlite3_value] object. An unprotected sqlite3_value object ** may only be used with [sqlite3_bind_value()] and [sqlite3_result_value()]. ** If the [unprotected sqlite3_value] object returned by @@ -3137,14 +3194,14 @@ ** and atof(). SQLite does not really use these functions. It has its ** own equivalent internal routines. The atoi() and atof() names are ** used in the table for brevity and because they are familiar to most ** C programmers. ** -** ^Note that when type conversions occur, pointers returned by prior +** Note that when type conversions occur, pointers returned by prior ** calls to sqlite3_column_blob(), sqlite3_column_text(), and/or ** sqlite3_column_text16() may be invalidated. -** ^(Type conversions and pointer invalidations might occur +** Type conversions and pointer invalidations might occur ** in the following cases: ** ** <ul> ** <li> The initial content is a BLOB and sqlite3_column_text() or ** sqlite3_column_text16() is called. A zero-terminator might @@ -3153,26 +3210,26 @@ ** sqlite3_column_text16() is called. The content must be converted ** to UTF-16.</li> ** <li> The initial content is UTF-16 text and sqlite3_column_bytes() or ** sqlite3_column_text() is called. The content must be converted ** to UTF-8.</li> -** </ul>)^ +** </ul> ** ** ^Conversions between UTF-16be and UTF-16le are always done in place and do ** not invalidate a prior pointer, though of course the content of the buffer -** that the prior pointer points to will have been modified. Other kinds +** that the prior pointer references will have been modified. Other kinds ** of conversion are done in place when it is possible, but sometimes they ** are not possible and in those cases prior pointers are invalidated. ** -** ^(The safest and easiest to remember policy is to invoke these routines +** The safest and easiest to remember policy is to invoke these routines ** in one of the following ways: ** ** <ul> ** <li>sqlite3_column_text() followed by sqlite3_column_bytes()</li> ** <li>sqlite3_column_blob() followed by sqlite3_column_bytes()</li> ** <li>sqlite3_column_text16() followed by sqlite3_column_bytes16()</li> -** </ul>)^ +** </ul> ** ** In other words, you should call sqlite3_column_text(), ** sqlite3_column_blob(), or sqlite3_column_text16() first to force the result ** into the desired format, then invoke sqlite3_column_bytes() or ** sqlite3_column_bytes16() to find the size of the result. Do not mix calls @@ -3206,21 +3263,30 @@ /* ** CAPI3REF: Destroy A Prepared Statement Object ** ** ^The sqlite3_finalize() function is called to delete a [prepared statement]. -** ^If the statement was executed successfully or not executed at all, then -** SQLITE_OK is returned. ^If execution of the statement failed then an -** [error code] or [extended error code] is returned. +** ^If the most recent evaluation of the statement encountered no errors or +** or if the statement is never been evaluated, then sqlite3_finalize() returns +** SQLITE_OK. ^If the most recent evaluation of statement S failed, then +** sqlite3_finalize(S) returns the appropriate [error code] or +** [extended error code]. ** -** ^This routine can be called at any point during the execution of the -** [prepared statement]. ^If the virtual machine has not -** completed execution when this routine is called, that is like -** encountering an error or an [sqlite3_interrupt | interrupt]. -** ^Incomplete updates may be rolled back and transactions canceled, -** depending on the circumstances, and the -** [error code] returned will be [SQLITE_ABORT]. +** ^The sqlite3_finalize(S) routine can be called at any point during +** the life cycle of [prepared statement] S: +** before statement S is ever evaluated, after +** one or more calls to [sqlite3_reset()], or after any call +** to [sqlite3_step()] regardless of whether or not the statement has +** completed execution. +** +** ^Invoking sqlite3_finalize() on a NULL pointer is a harmless no-op. +** +** The application must finalize every [prepared statement] in order to avoid +** resource leaks. It is a grievous error for the application to try to use +** a prepared statement after it has been finalized. Any use of a prepared +** statement after it has been finalized can result in undefined and +** undesirable behavior such as segfaults and heap corruption. */ SQLITE_API int sqlite3_finalize(sqlite3_stmt *pStmt); /* ** CAPI3REF: Reset A Prepared Statement Object @@ -3252,40 +3318,42 @@ ** CAPI3REF: Create Or Redefine SQL Functions ** KEYWORDS: {function creation routines} ** KEYWORDS: {application-defined SQL function} ** KEYWORDS: {application-defined SQL functions} ** -** ^These two functions (collectively known as "function creation routines") +** ^These functions (collectively known as "function creation routines") ** are used to add SQL functions or aggregates or to redefine the behavior -** of existing SQL functions or aggregates. The only difference between the -** two is that the second parameter, the name of the (scalar) function or -** aggregate, is encoded in UTF-8 for sqlite3_create_function() and UTF-16 -** for sqlite3_create_function16(). +** of existing SQL functions or aggregates. The only differences between +** these routines are the text encoding expected for +** the the second parameter (the name of the function being created) +** and the presence or absence of a destructor callback for +** the application data pointer. ** ** ^The first parameter is the [database connection] to which the SQL ** function is to be added. ^If an application uses more than one database ** connection then application-defined SQL functions must be added ** to each database connection separately. ** -** The second parameter is the name of the SQL function to be created or -** redefined. ^The length of the name is limited to 255 bytes, exclusive of -** the zero-terminator. Note that the name length limit is in bytes, not -** characters. ^Any attempt to create a function with a longer name -** will result in [SQLITE_ERROR] being returned. +** ^The second parameter is the name of the SQL function to be created or +** redefined. ^The length of the name is limited to 255 bytes in a UTF-8 +** representation, exclusive of the zero-terminator. ^Note that the name +** length limit is in UTF-8 bytes, not characters nor UTF-16 bytes. +** ^Any attempt to create a function with a longer name +** will result in [SQLITE_MISUSE] being returned. ** ** ^The third parameter (nArg) ** is the number of arguments that the SQL function or ** aggregate takes. ^If this parameter is -1, then the SQL function or ** aggregate may take any number of arguments between 0 and the limit ** set by [sqlite3_limit]([SQLITE_LIMIT_FUNCTION_ARG]). If the third ** parameter is less than -1 or greater than 127 then the behavior is ** undefined. ** -** The fourth parameter, eTextRep, specifies what +** ^The fourth parameter, eTextRep, specifies what ** [SQLITE_UTF8 | text encoding] this SQL function prefers for -** its parameters. Any SQL function implementation should be able to work -** work with UTF-8, UTF-16le, or UTF-16be. But some implementations may be +** its parameters. Every SQL function implementation must be able to work +** with UTF-8, UTF-16le, or UTF-16be. But some implementations may be ** more efficient with one encoding than another. ^An application may ** invoke sqlite3_create_function() or sqlite3_create_function16() multiple ** times with the same function but with different values of eTextRep. ** ^When multiple implementations of the same function are available, SQLite ** will pick the one that involves the least amount of data conversion. @@ -3293,17 +3361,25 @@ ** encoding is used, then the fourth argument should be [SQLITE_ANY]. ** ** ^(The fifth parameter is an arbitrary pointer. The implementation of the ** function can gain access to this pointer using [sqlite3_user_data()].)^ ** -** The seventh, eighth and ninth parameters, xFunc, xStep and xFinal, are +** ^The seventh, eighth and ninth parameters, xFunc, xStep and xFinal, are ** pointers to C-language functions that implement the SQL function or ** aggregate. ^A scalar SQL function requires an implementation of the xFunc -** callback only; NULL pointers should be passed as the xStep and xFinal +** callback only; NULL pointers must be passed as the xStep and xFinal ** parameters. ^An aggregate SQL function requires an implementation of xStep -** and xFinal and NULL should be passed for xFunc. ^To delete an existing -** SQL function or aggregate, pass NULL for all three function callbacks. +** and xFinal and NULL pointer must be passed for xFunc. ^To delete an existing +** SQL function or aggregate, pass NULL poiners for all three function +** callbacks. +** +** ^If the tenth parameter to sqlite3_create_function_v2() is not NULL, +** then it is invoked when the function is deleted, either by being +** overloaded or when the database connection closes. +** ^When the destructure callback of the tenth parameter is invoked, it +** is passed a single argument which is a copy of the pointer which was +** the fifth parameter to sqlite3_create_function_v2(). ** ** ^It is permitted to register multiple implementations of the same ** functions with the same name but with either differing numbers of ** arguments or differing preferred text encodings. ^SQLite will use ** the implementation that most closely matches the way in which the @@ -3315,15 +3391,10 @@ ** ^A function where the encoding difference is between UTF16le and UTF16be ** is a closer match than a function where the encoding difference is ** between UTF8 and UTF16. ** ** ^Built-in functions may be overloaded by new application-defined functions. -** ^The first application-defined function with a given name overrides all -** built-in functions in the same [database connection] with the same name. -** ^Subsequent application-defined functions of the same name only override -** prior application-defined functions that are an exact match for the -** number of parameters and preferred encoding. ** ** ^An application-defined function is permitted to call other ** SQLite interfaces. However, such calls must not ** close the database connection nor finalize or reset the prepared ** statement in which the function is running. @@ -3345,10 +3416,21 @@ int eTextRep, void *pApp, void (*xFunc)(sqlite3_context*,int,sqlite3_value**), void (*xStep)(sqlite3_context*,int,sqlite3_value**), void (*xFinal)(sqlite3_context*) +); +SQLITE_API int sqlite3_create_function_v2( + sqlite3 *db, + const char *zFunctionName, + int nArg, + int eTextRep, + void *pApp, + void (*xFunc)(sqlite3_context*,int,sqlite3_value**), + void (*xStep)(sqlite3_context*,int,sqlite3_value**), + void (*xFinal)(sqlite3_context*), + void(*xDestroy)(void*) ); /* ** CAPI3REF: Text Encodings ** @@ -3440,11 +3522,11 @@ SQLITE_API int sqlite3_value_numeric_type(sqlite3_value*); /* ** CAPI3REF: Obtain Aggregate Function Context ** -** Implementions of aggregate SQL functions use this +** Implementations of aggregate SQL functions use this ** routine to allocate memory for storing their state. ** ** ^The first time the sqlite3_aggregate_context(C,N) routine is called ** for a particular aggregate function, SQLite ** allocates N of memory, zeroes out that memory, and returns a pointer @@ -3692,73 +3774,97 @@ SQLITE_API void sqlite3_result_zeroblob(sqlite3_context*, int n); /* ** CAPI3REF: Define New Collating Sequences ** -** These functions are used to add new collation sequences to the -** [database connection] specified as the first argument. +** ^These functions add, remove, or modify a [collation] associated +** with the [database connection] specified as the first argument. ** -** ^The name of the new collation sequence is specified as a UTF-8 string +** ^The name of the collation is a UTF-8 string ** for sqlite3_create_collation() and sqlite3_create_collation_v2() -** and a UTF-16 string for sqlite3_create_collation16(). ^In all cases -** the name is passed as the second function argument. -** -** ^The third argument may be one of the constants [SQLITE_UTF8], -** [SQLITE_UTF16LE], or [SQLITE_UTF16BE], indicating that the user-supplied -** routine expects to be passed pointers to strings encoded using UTF-8, -** UTF-16 little-endian, or UTF-16 big-endian, respectively. ^The -** third argument might also be [SQLITE_UTF16] to indicate that the routine -** expects pointers to be UTF-16 strings in the native byte order, or the -** argument can be [SQLITE_UTF16_ALIGNED] if the -** the routine expects pointers to 16-bit word aligned strings -** of UTF-16 in the native byte order. -** -** A pointer to the user supplied routine must be passed as the fifth -** argument. ^If it is NULL, this is the same as deleting the collation -** sequence (so that SQLite cannot call it anymore). -** ^Each time the application supplied function is invoked, it is passed -** as its first parameter a copy of the void* passed as the fourth argument -** to sqlite3_create_collation() or sqlite3_create_collation16(). -** -** ^The remaining arguments to the application-supplied routine are two strings, -** each represented by a (length, data) pair and encoded in the encoding -** that was passed as the third argument when the collation sequence was -** registered. The application defined collation routine should -** return negative, zero or positive if the first string is less than, -** equal to, or greater than the second string. i.e. (STRING1 - STRING2). +** and a UTF-16 string in native byte order for sqlite3_create_collation16(). +** ^Collation names that compare equal according to [sqlite3_strnicmp()] are +** considered to be the same name. +** +** ^(The third argument (eTextRep) must be one of the constants: +** <ul> +** <li> [SQLITE_UTF8], +** <li> [SQLITE_UTF16LE], +** <li> [SQLITE_UTF16BE], +** <li> [SQLITE_UTF16], or +** <li> [SQLITE_UTF16_ALIGNED]. +** </ul>)^ +** ^The eTextRep argument determines the encoding of strings passed +** to the collating function callback, xCallback. +** ^The [SQLITE_UTF16] and [SQLITE_UTF16_ALIGNED] values for eTextRep +** force strings to be UTF16 with native byte order. +** ^The [SQLITE_UTF16_ALIGNED] value for eTextRep forces strings to begin +** on an even byte address. +** +** ^The fourth argument, pArg, is a application data pointer that is passed +** through as the first argument to the collating function callback. +** +** ^The fifth argument, xCallback, is a pointer to the collating function. +** ^Multiple collating functions can be registered using the same name but +** with different eTextRep parameters and SQLite will use whichever +** function requires the least amount of data transformation. +** ^If the xCallback argument is NULL then the collating function is +** deleted. ^When all collating functions having the same name are deleted, +** that collation is no longer usable. +** +** ^The collating function callback is invoked with a copy of the pArg +** application data pointer and with two strings in the encoding specified +** by the eTextRep argument. The collating function must return an +** integer that is negative, zero, or positive +** if the first string is less than, equal to, or greater than the second, +** respectively. A collating function must alway return the same answer +** given the same inputs. If two or more collating functions are registered +** to the same collation name (using different eTextRep values) then all +** must give an equivalent answer when invoked with equivalent strings. +** The collating function must obey the following properties for all +** strings A, B, and C: +** +** <ol> +** <li> If A==B then B==A. +** <li> If A==B and B==C then A==C. +** <li> If A<B THEN B>A. +** <li> If A<B and B<C then A<C. +** </ol> +** +** If a collating function fails any of the above constraints and that +** collating function is registered and used, then the behavior of SQLite +** is undefined. ** ** ^The sqlite3_create_collation_v2() works like sqlite3_create_collation() -** except that it takes an extra argument which is a destructor for -** the collation. ^The destructor is called when the collation is -** destroyed and is passed a copy of the fourth parameter void* pointer -** of the sqlite3_create_collation_v2(). -** ^Collations are destroyed when they are overridden by later calls to the -** collation creation functions or when the [database connection] is closed -** using [sqlite3_close()]. +** with the addition that the xDestroy callback is invoked on pArg when +** the collating function is deleted. +** ^Collating functions are deleted when they are overridden by later +** calls to the collation creation functions or when the +** [database connection] is closed using [sqlite3_close()]. ** ** See also: [sqlite3_collation_needed()] and [sqlite3_collation_needed16()]. */ SQLITE_API int sqlite3_create_collation( sqlite3*, const char *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*) ); SQLITE_API int sqlite3_create_collation_v2( sqlite3*, const char *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*), void(*xDestroy)(void*) ); SQLITE_API int sqlite3_create_collation16( sqlite3*, const void *zName, int eTextRep, - void*, + void *pArg, int(*xCompare)(void*,int,const void*,int,const void*) ); /* ** CAPI3REF: Collation Needed Callbacks @@ -3843,20 +3949,23 @@ #endif /* ** CAPI3REF: Suspend Execution For A Short Time ** -** ^The sqlite3_sleep() function causes the current thread to suspend execution +** The sqlite3_sleep() function causes the current thread to suspend execution ** for at least a number of milliseconds specified in its parameter. ** -** ^If the operating system does not support sleep requests with +** If the operating system does not support sleep requests with ** millisecond time resolution, then the time will be rounded up to -** the nearest second. ^The number of milliseconds of sleep actually +** the nearest second. The number of milliseconds of sleep actually ** requested from the operating system is returned. ** ** ^SQLite implements this interface by calling the xSleep() -** method of the default [sqlite3_vfs] object. +** method of the default [sqlite3_vfs] object. If the xSleep() method +** of the default VFS is not implemented correctly, or not implemented at +** all, then the behavior of sqlite3_sleep() may deviate from the description +** in the previous paragraphs. */ SQLITE_API int sqlite3_sleep(int); /* ** CAPI3REF: Name Of The Folder Holding Temporary Files @@ -4074,44 +4183,77 @@ ** of heap memory by deallocating non-essential memory allocations ** held by the database library. Memory used to cache database ** pages to improve performance is an example of non-essential memory. ** ^sqlite3_release_memory() returns the number of bytes actually freed, ** which might be more or less than the amount requested. +** ^The sqlite3_release_memory() routine is a no-op returning zero +** if SQLite is not compiled with [SQLITE_ENABLE_MEMORY_MANAGEMENT]. */ SQLITE_API int sqlite3_release_memory(int); /* ** CAPI3REF: Impose A Limit On Heap Size ** -** ^The sqlite3_soft_heap_limit() interface places a "soft" limit -** on the amount of heap memory that may be allocated by SQLite. -** ^If an internal allocation is requested that would exceed the -** soft heap limit, [sqlite3_release_memory()] is invoked one or -** more times to free up some space before the allocation is performed. -** -** ^The limit is called "soft" because if [sqlite3_release_memory()] -** cannot free sufficient memory to prevent the limit from being exceeded, -** the memory is allocated anyway and the current operation proceeds. -** -** ^A negative or zero value for N means that there is no soft heap limit and -** [sqlite3_release_memory()] will only be called when memory is exhausted. -** ^The default value for the soft heap limit is zero. -** -** ^(SQLite makes a best effort to honor the soft heap limit. -** But if the soft heap limit cannot be honored, execution will -** continue without error or notification.)^ This is why the limit is -** called a "soft" limit. It is advisory only. -** -** Prior to SQLite version 3.5.0, this routine only constrained the memory -** allocated by a single thread - the same thread in which this routine -** runs. Beginning with SQLite version 3.5.0, the soft heap limit is -** applied to all threads. The value specified for the soft heap limit -** is an upper bound on the total memory allocation for all threads. In -** version 3.5.0 there is no mechanism for limiting the heap usage for -** individual threads. +** ^The sqlite3_soft_heap_limit64() interface sets and/or queries the +** soft limit on the amount of heap memory that may be allocated by SQLite. +** ^SQLite strives to keep heap memory utilization below the soft heap +** limit by reducing the number of pages held in the page cache +** as heap memory usages approaches the limit. +** ^The soft heap limit is "soft" because even though SQLite strives to stay +** below the limit, it will exceed the limit rather than generate +** an [SQLITE_NOMEM] error. In other words, the soft heap limit +** is advisory only. +** +** ^The return value from sqlite3_soft_heap_limit64() is the size of +** the soft heap limit prior to the call. ^If the argument N is negative +** then no change is made to the soft heap limit. Hence, the current +** size of the soft heap limit can be determined by invoking +** sqlite3_soft_heap_limit64() with a negative argument. +** +** ^If the argument N is zero then the soft heap limit is disabled. +** +** ^(The soft heap limit is not enforced in the current implementation +** if one or more of following conditions are true: +** +** <ul> +** <li> The soft heap limit is set to zero. +** <li> Memory accounting is disabled using a combination of the +** [sqlite3_config]([SQLITE_CONFIG_MEMSTATUS],...) start-time option and +** the [SQLITE_DEFAULT_MEMSTATUS] compile-time option. +** <li> An alternative page cache implementation is specifed using +** [sqlite3_config]([SQLITE_CONFIG_PCACHE],...). +** <li> The page cache allocates from its own memory pool supplied +** by [sqlite3_config]([SQLITE_CONFIG_PAGECACHE],...) rather than +** from the heap. +** </ul>)^ +** +** Beginning with SQLite version 3.7.3, the soft heap limit is enforced +** regardless of whether or not the [SQLITE_ENABLE_MEMORY_MANAGEMENT] +** compile-time option is invoked. With [SQLITE_ENABLE_MEMORY_MANAGEMENT], +** the soft heap limit is enforced on every memory allocation. Without +** [SQLITE_ENABLE_MEMORY_MANAGEMENT], the soft heap limit is only enforced +** when memory is allocated by the page cache. Testing suggests that because +** the page cache is the predominate memory user in SQLite, most +** applications will achieve adequate soft heap limit enforcement without +** the use of [SQLITE_ENABLE_MEMORY_MANAGEMENT]. +** +** The circumstances under which SQLite will enforce the soft heap limit may +** changes in future releases of SQLite. +*/ +SQLITE_API sqlite3_int64 sqlite3_soft_heap_limit64(sqlite3_int64 N); + +/* +** CAPI3REF: Deprecated Soft Heap Limit Interface +** DEPRECATED +** +** This is a deprecated version of the [sqlite3_soft_heap_limit64()] +** interface. This routine is provided for historical compatibility +** only. All new applications should use the +** [sqlite3_soft_heap_limit64()] interface rather than this one. */ -SQLITE_API void sqlite3_soft_heap_limit(int); +SQLITE_API SQLITE_DEPRECATED void sqlite3_soft_heap_limit(int N); + /* ** CAPI3REF: Extract Metadata About A Column Of A Table ** ** ^This routine returns metadata about a specific column of a specific @@ -4231,38 +4373,51 @@ ** it back off again. */ SQLITE_API int sqlite3_enable_load_extension(sqlite3 *db, int onoff); /* -** CAPI3REF: Automatically Load An Extensions -** -** ^This API can be invoked at program startup in order to register -** one or more statically linked extensions that will be available -** to all new [database connections]. -** -** ^(This routine stores a pointer to the extension entry point -** in an array that is obtained from [sqlite3_malloc()]. That memory -** is deallocated by [sqlite3_reset_auto_extension()].)^ -** -** ^This function registers an extension entry point that is -** automatically invoked whenever a new [database connection] -** is opened using [sqlite3_open()], [sqlite3_open16()], -** or [sqlite3_open_v2()]. -** ^Duplicate extensions are detected so calling this routine -** multiple times with the same extension is harmless. -** ^Automatic extensions apply across all threads. +** CAPI3REF: Automatically Load Statically Linked Extensions +** +** ^This interface causes the xEntryPoint() function to be invoked for +** each new [database connection] that is created. The idea here is that +** xEntryPoint() is the entry point for a statically linked SQLite extension +** that is to be automatically loaded into all new database connections. +** +** ^(Even though the function prototype shows that xEntryPoint() takes +** no arguments and returns void, SQLite invokes xEntryPoint() with three +** arguments and expects and integer result as if the signature of the +** entry point where as follows: +** +** <blockquote><pre> +** int xEntryPoint( +** sqlite3 *db, +** const char **pzErrMsg, +** const struct sqlite3_api_routines *pThunk +** ); +** </pre></blockquote>)^ +** +** If the xEntryPoint routine encounters an error, it should make *pzErrMsg +** point to an appropriate error message (obtained from [sqlite3_mprintf()]) +** and return an appropriate [error code]. ^SQLite ensures that *pzErrMsg +** is NULL before calling the xEntryPoint(). ^SQLite will invoke +** [sqlite3_free()] on *pzErrMsg after xEntryPoint() returns. ^If any +** xEntryPoint() returns an error, the [sqlite3_open()], [sqlite3_open16()], +** or [sqlite3_open_v2()] call that provoked the xEntryPoint() will fail. +** +** ^Calling sqlite3_auto_extension(X) with an entry point X that is already +** on the list of automatic extensions is a harmless no-op. ^No entry point +** will be called more than once for each database connection that is opened. +** +** See also: [sqlite3_reset_auto_extension()]. */ SQLITE_API int sqlite3_auto_extension(void (*xEntryPoint)(void)); /* ** CAPI3REF: Reset Automatic Extension Loading ** -** ^(This function disables all previously registered automatic -** extensions. It undoes the effect of all prior -** [sqlite3_auto_extension()] calls.)^ -** -** ^This function disables automatic extensions in all threads. +** ^This interface disables all automatic extensions previously +** registered using [sqlite3_auto_extension()]. */ SQLITE_API void sqlite3_reset_auto_extension(void); /* ** The interface to the virtual-table mechanism is currently considered @@ -4897,11 +5052,11 @@ ** output variable when querying the system for the current mutex ** implementation, using the [SQLITE_CONFIG_GETMUTEX] option. ** ** ^The xMutexInit method defined by this structure is invoked as ** part of system initialization by the sqlite3_initialize() function. -** ^The xMutexInit routine is calle by SQLite exactly once for each +** ^The xMutexInit routine is called by SQLite exactly once for each ** effective call to [sqlite3_initialize()]. ** ** ^The xMutexEnd method defined by this structure is invoked as ** part of system shutdown by the sqlite3_shutdown() function. The ** implementation of this method is expected to release all outstanding @@ -4930,11 +5085,11 @@ ** of passing a NULL pointer instead of a valid mutex handle are undefined ** (i.e. it is acceptable to provide an implementation that segfaults if ** it is passed a NULL pointer). ** ** The xMutexInit() method must be threadsafe. ^It must be harmless to -** invoke xMutexInit() mutiple times within the same process and without +** invoke xMutexInit() multiple times within the same process and without ** intervening calls to xMutexEnd(). Second and subsequent calls to ** xMutexInit() must be no-ops. ** ** ^xMutexInit() must not use SQLite memory allocation ([sqlite3_malloc()] ** and its associates). ^Similarly, xMutexAlloc() must not use SQLite memory @@ -5094,17 +5249,18 @@ #define SQLITE_TESTCTRL_ALWAYS 13 #define SQLITE_TESTCTRL_RESERVE 14 #define SQLITE_TESTCTRL_OPTIMIZATIONS 15 #define SQLITE_TESTCTRL_ISKEYWORD 16 #define SQLITE_TESTCTRL_PGHDRSZ 17 -#define SQLITE_TESTCTRL_LAST 17 +#define SQLITE_TESTCTRL_SCRATCHMALLOC 18 +#define SQLITE_TESTCTRL_LAST 18 /* ** CAPI3REF: SQLite Runtime Status ** ** ^This interface is used to retrieve runtime status information -** about the preformance of SQLite, and optionally to reset various +** about the performance of SQLite, and optionally to reset various ** highwater marks. ^The first argument is an integer code for ** the specific parameter to measure. ^(Recognized integer codes ** are of the form [SQLITE_STATUS_MEMORY_USED | SQLITE_STATUS_...].)^ ** ^The current value of the parameter is returned into *pCurrent. ** ^The highest recorded value is returned in *pHighwater. ^If the @@ -5113,11 +5269,11 @@ ** value. For those parameters ** nothing is written into *pHighwater and the resetFlag is ignored.)^ ** ^(Other parameters record only the highwater mark and not the current ** value. For these latter parameters nothing is written into *pCurrent.)^ ** -** ^The sqlite3_db_status() routine returns SQLITE_OK on success and a +** ^The sqlite3_status() routine returns SQLITE_OK on success and a ** non-zero [error code] on failure. ** ** This routine is threadsafe but is not atomic. This routine can be ** called while other threads are running the same or different SQLite ** interfaces. However the values returned in *pCurrent and @@ -5163,11 +5319,11 @@ ** [SQLITE_CONFIG_PAGECACHE]. The ** value returned is in pages, not in bytes.</dd>)^ ** ** ^(<dt>SQLITE_STATUS_PAGECACHE_OVERFLOW</dt> ** <dd>This parameter returns the number of bytes of page cache -** allocation which could not be statisfied by the [SQLITE_CONFIG_PAGECACHE] +** allocation which could not be satisfied by the [SQLITE_CONFIG_PAGECACHE] ** buffer and where forced to overflow to [sqlite3_malloc()]. The ** returned value includes allocations that overflowed because they ** where too large (they were larger than the "sz" parameter to ** [SQLITE_CONFIG_PAGECACHE]) and allocations that overflowed because ** no space was left in the page cache.</dd>)^ @@ -5186,11 +5342,11 @@ ** outstanding at time, this parameter also reports the number of threads ** using scratch memory at the same time.</dd>)^ ** ** ^(<dt>SQLITE_STATUS_SCRATCH_OVERFLOW</dt> ** <dd>This parameter returns the number of bytes of scratch memory -** allocation which could not be statisfied by the [SQLITE_CONFIG_SCRATCH] +** allocation which could not be satisfied by the [SQLITE_CONFIG_SCRATCH] ** buffer and where forced to overflow to [sqlite3_malloc()]. The values ** returned include overflows because the requested allocation was too ** larger (that is, because the requested allocation was larger than the ** "sz" parameter to [SQLITE_CONFIG_SCRATCH]) and because no scratch buffer ** slots were available. @@ -5226,18 +5382,21 @@ ** ^This interface is used to retrieve runtime status information ** about a single [database connection]. ^The first argument is the ** database connection object to be interrogated. ^The second argument ** is an integer constant, taken from the set of ** [SQLITE_DBSTATUS_LOOKASIDE_USED | SQLITE_DBSTATUS_*] macros, that -** determiness the parameter to interrogate. The set of +** determines the parameter to interrogate. The set of ** [SQLITE_DBSTATUS_LOOKASIDE_USED | SQLITE_DBSTATUS_*] macros is likely ** to grow in future releases of SQLite. ** ** ^The current value of the requested parameter is written into *pCur ** and the highest instantaneous value is written into *pHiwtr. ^If ** the resetFlg is true, then the highest instantaneous value is ** reset back down to the current value. +** +** ^The sqlite3_db_status() routine returns SQLITE_OK on success and a +** non-zero [error code] on failure. ** ** See also: [sqlite3_status()] and [sqlite3_stmt_status()]. */ SQLITE_API int sqlite3_db_status(sqlite3*, int op, int *pCur, int *pHiwtr, int resetFlg); @@ -5361,122 +5520,134 @@ ** CAPI3REF: Application Defined Page Cache. ** KEYWORDS: {page cache} ** ** ^(The [sqlite3_config]([SQLITE_CONFIG_PCACHE], ...) interface can ** register an alternative page cache implementation by passing in an -** instance of the sqlite3_pcache_methods structure.)^ The majority of the -** heap memory used by SQLite is used by the page cache to cache data read -** from, or ready to be written to, the database file. By implementing a -** custom page cache using this API, an application can control more -** precisely the amount of memory consumed by SQLite, the way in which +** instance of the sqlite3_pcache_methods structure.)^ +** In many applications, most of the heap memory allocated by +** SQLite is used for the page cache. +** By implementing a +** custom page cache using this API, an application can better control +** the amount of memory consumed by SQLite, the way in which ** that memory is allocated and released, and the policies used to ** determine exactly which parts of a database file are cached and for ** how long. ** +** The alternative page cache mechanism is an +** extreme measure that is only needed by the most demanding applications. +** The built-in page cache is recommended for most uses. +** ** ^(The contents of the sqlite3_pcache_methods structure are copied to an ** internal buffer by SQLite within the call to [sqlite3_config]. Hence ** the application may discard the parameter after the call to ** [sqlite3_config()] returns.)^ ** -** ^The xInit() method is called once for each call to [sqlite3_initialize()] +** ^(The xInit() method is called once for each effective +** call to [sqlite3_initialize()])^ ** (usually only once during the lifetime of the process). ^(The xInit() ** method is passed a copy of the sqlite3_pcache_methods.pArg value.)^ -** ^The xInit() method can set up up global structures and/or any mutexes +** The intent of the xInit() method is to set up global data structures ** required by the custom page cache implementation. +** ^(If the xInit() method is NULL, then the +** built-in default page cache is used instead of the application defined +** page cache.)^ ** -** ^The xShutdown() method is called from within [sqlite3_shutdown()], -** if the application invokes this API. It can be used to clean up +** ^The xShutdown() method is called by [sqlite3_shutdown()]. +** It can be used to clean up ** any outstanding resources before process shutdown, if required. +** ^The xShutdown() method may be NULL. ** -** ^SQLite holds a [SQLITE_MUTEX_RECURSIVE] mutex when it invokes -** the xInit method, so the xInit method need not be threadsafe. ^The +** ^SQLite automatically serializes calls to the xInit method, +** so the xInit method need not be threadsafe. ^The ** xShutdown method is only called from [sqlite3_shutdown()] so it does ** not need to be threadsafe either. All other methods must be threadsafe ** in multithreaded applications. ** ** ^SQLite will never invoke xInit() more than once without an intervening ** call to xShutdown(). ** -** ^The xCreate() method is used to construct a new cache instance. SQLite -** will typically create one cache instance for each open database file, +** ^SQLite invokes the xCreate() method to construct a new cache instance. +** SQLite will typically create one cache instance for each open database file, ** though this is not guaranteed. ^The ** first parameter, szPage, is the size in bytes of the pages that must ** be allocated by the cache. ^szPage will not be a power of two. ^szPage ** will the page size of the database file that is to be cached plus an -** increment (here called "R") of about 100 or 200. ^SQLite will use the +** increment (here called "R") of about 100 or 200. SQLite will use the ** extra R bytes on each page to store metadata about the underlying ** database page on disk. The value of R depends ** on the SQLite version, the target platform, and how SQLite was compiled. ** ^R is constant for a particular build of SQLite. ^The second argument to ** xCreate(), bPurgeable, is true if the cache being created will ** be used to cache database pages of a file stored on disk, or -** false if it is used for an in-memory database. ^The cache implementation +** false if it is used for an in-memory database. The cache implementation ** does not have to do anything special based with the value of bPurgeable; ** it is purely advisory. ^On a cache where bPurgeable is false, SQLite will ** never invoke xUnpin() except to deliberately delete a page. -** ^In other words, a cache created with bPurgeable set to false will +** ^In other words, calls to xUnpin() on a cache with bPurgeable set to +** false will always have the "discard" flag set to true. +** ^Hence, a cache created with bPurgeable false will ** never contain any unpinned pages. ** ** ^(The xCachesize() method may be called at any time by SQLite to set the ** suggested maximum cache-size (number of pages stored by) the cache ** instance passed as the first argument. This is the value configured using -** the SQLite "[PRAGMA cache_size]" command.)^ ^As with the bPurgeable +** the SQLite "[PRAGMA cache_size]" command.)^ As with the bPurgeable ** parameter, the implementation is not required to do anything with this ** value; it is advisory only. ** -** ^The xPagecount() method should return the number of pages currently -** stored in the cache. +** The xPagecount() method must return the number of pages currently +** stored in the cache, both pinned and unpinned. ** -** ^The xFetch() method is used to fetch a page and return a pointer to it. -** ^A 'page', in this context, is a buffer of szPage bytes aligned at an -** 8-byte boundary. ^The page to be fetched is determined by the key. ^The -** mimimum key value is 1. After it has been retrieved using xFetch, the page +** The xFetch() method locates a page in the cache and returns a pointer to +** the page, or a NULL pointer. +** A "page", in this context, means a buffer of szPage bytes aligned at an +** 8-byte boundary. The page to be fetched is determined by the key. ^The +** mimimum key value is 1. After it has been retrieved using xFetch, the page ** is considered to be "pinned". ** -** ^If the requested page is already in the page cache, then the page cache +** If the requested page is already in the page cache, then the page cache ** implementation must return a pointer to the page buffer with its content -** intact. ^(If the requested page is not already in the cache, then the -** behavior of the cache implementation is determined by the value of the -** createFlag parameter passed to xFetch, according to the following table: +** intact. If the requested page is not already in the cache, then the +** behavior of the cache implementation should use the value of the createFlag +** parameter to help it determined what action to take: ** ** <table border=1 width=85% align=center> ** <tr><th> createFlag <th> Behaviour when page is not already in cache ** <tr><td> 0 <td> Do not allocate a new page. Return NULL. ** <tr><td> 1 <td> Allocate a new page if it easy and convenient to do so. ** Otherwise return NULL. ** <tr><td> 2 <td> Make every effort to allocate a new page. Only return ** NULL if allocating a new page is effectively impossible. -** </table>)^ +** </table> ** -** SQLite will normally invoke xFetch() with a createFlag of 0 or 1. If -** a call to xFetch() with createFlag==1 returns NULL, then SQLite will +** ^(SQLite will normally invoke xFetch() with a createFlag of 0 or 1. SQLite +** will only use a createFlag of 2 after a prior call with a createFlag of 1 +** failed.)^ In between the to xFetch() calls, SQLite may ** attempt to unpin one or more cache pages by spilling the content of -** pinned pages to disk and synching the operating system disk cache. After -** attempting to unpin pages, the xFetch() method will be invoked again with -** a createFlag of 2. +** pinned pages to disk and synching the operating system disk cache. ** ** ^xUnpin() is called by SQLite with a pointer to a currently pinned page -** as its second argument. ^(If the third parameter, discard, is non-zero, -** then the page should be evicted from the cache. In this case SQLite -** assumes that the next time the page is retrieved from the cache using -** the xFetch() method, it will be zeroed.)^ ^If the discard parameter is -** zero, then the page is considered to be unpinned. ^The cache implementation +** as its second argument. If the third parameter, discard, is non-zero, +** then the page must be evicted from the cache. +** ^If the discard parameter is +** zero, then the page may be discarded or retained at the discretion of +** page cache implementation. ^The page cache implementation ** may choose to evict unpinned pages at any time. ** -** ^(The cache is not required to perform any reference counting. A single +** The cache must not perform any reference counting. A single ** call to xUnpin() unpins the page regardless of the number of prior calls -** to xFetch().)^ +** to xFetch(). ** -** ^The xRekey() method is used to change the key value associated with the -** page passed as the second argument from oldKey to newKey. ^If the cache -** previously contains an entry associated with newKey, it should be +** The xRekey() method is used to change the key value associated with the +** page passed as the second argument. If the cache +** previously contains an entry associated with newKey, it must be ** discarded. ^Any prior cache entry associated with newKey is guaranteed not ** to be pinned. ** -** ^When SQLite calls the xTruncate() method, the cache must discard all +** When SQLite calls the xTruncate() method, the cache must discard all ** existing cache entries with page numbers (keys) greater than or equal -** to the value of the iLimit parameter passed to xTruncate(). ^If any +** to the value of the iLimit parameter passed to xTruncate(). If any ** of these pages are pinned, they are implicitly unpinned, meaning that ** they can be safely discarded. ** ** ^The xDestroy() method is used to delete a cache allocated by xCreate(). ** All resources associated with the specified cache should be freed. ^After @@ -5648,11 +5819,11 @@ ** ** <b>sqlite3_backup_remaining(), sqlite3_backup_pagecount()</b> ** ** ^Each call to sqlite3_backup_step() sets two values inside ** the [sqlite3_backup] object: the number of pages still to be backed -** up and the total number of pages in the source databae file. +** up and the total number of pages in the source database file. ** The sqlite3_backup_remaining() and sqlite3_backup_pagecount() interfaces ** retrieve these two values, respectively. ** ** ^The values returned by these functions are only updated by ** sqlite3_backup_step(). ^If the source database is modified during a backup @@ -5744,11 +5915,11 @@ ** ^(There may be at most one unlock-notify callback registered by a ** blocked connection. If sqlite3_unlock_notify() is called when the ** blocked connection already has a registered unlock-notify callback, ** then the new callback replaces the old.)^ ^If sqlite3_unlock_notify() is ** called with a NULL pointer as its second argument, then any existing -** unlock-notify callback is cancelled. ^The blocked connections +** unlock-notify callback is canceled. ^The blocked connections ** unlock-notify callback may also be canceled by closing the blocked ** connection using [sqlite3_close()]. ** ** The unlock-notify callback is not reentrant. If an application invokes ** any sqlite3_xxx API functions from within an unlock-notify callback, a @@ -5826,11 +5997,11 @@ /* ** CAPI3REF: String Comparison ** ** ^The [sqlite3_strnicmp()] API allows applications and extensions to ** compare the contents of two buffers containing UTF-8 strings in a -** case-indendent fashion, using the same definition of case independence +** case-independent fashion, using the same definition of case independence ** that SQLite uses internally when comparing identifiers. */ SQLITE_API int sqlite3_strnicmp(const char *, const char *, int); /* @@ -5950,5 +6121,61 @@ #ifdef __cplusplus } /* End of the 'extern "C"' block */ #endif #endif +/* +** 2010 August 30 +** +** The author disclaims copyright to this source code. In place of +** a legal notice, here is a blessing: +** +** May you do good and not evil. +** May you find forgiveness for yourself and forgive others. +** May you share freely, never taking more than you give. +** +************************************************************************* +*/ + +#ifndef _SQLITE3RTREE_H_ +#define _SQLITE3RTREE_H_ + + +#ifdef __cplusplus +extern "C" { +#endif + +typedef struct sqlite3_rtree_geometry sqlite3_rtree_geometry; + +/* +** Register a geometry callback named zGeom that can be used as part of an +** R-Tree geometry query as follows: +** +** SELECT ... FROM <rtree> WHERE <rtree col> MATCH $zGeom(... params ...) +*/ +SQLITE_API int sqlite3_rtree_geometry_callback( + sqlite3 *db, + const char *zGeom, + int (*xGeom)(sqlite3_rtree_geometry *, int nCoord, double *aCoord, int *pRes), + void *pContext +); + + +/* +** A pointer to a structure of the following type is passed as the first +** argument to callbacks registered using rtree_geometry_callback(). +*/ +struct sqlite3_rtree_geometry { + void *pContext; /* Copy of pContext passed to s_r_g_c() */ + int nParam; /* Size of array aParam[] */ + double *aParam; /* Parameters passed to SQL geom function */ + void *pUser; /* Callback implementation user data */ + void (*xDelUser)(void *); /* Called by SQLite to clean up pUser */ +}; + + +#ifdef __cplusplus +} /* end of the 'extern "C"' block */ +#endif + +#endif /* ifndef _SQLITE3RTREE_H_ */ + Index: src/stat.c ================================================================== --- src/stat.c +++ src/stat.c @@ -26,32 +26,42 @@ ** WEBPAGE: stat ** ** Show statistics and global information about the repository. */ void stat_page(void){ - i64 t; - int n, m, fsize; + i64 t, fsize; + int n, m; + int szMax, szAvg; char zBuf[100]; + login_check_credentials(); if( !g.okRead ){ login_needed(); return; } style_header("Repository Statistics"); - @ <p><table class="label-value"> + @ <table class="label-value"> @ <tr><th>Repository Size:</th><td> fsize = file_size(g.zRepositoryName); - @ %d(fsize) bytes + sqlite3_snprintf(sizeof(zBuf), zBuf, "%lld", fsize); + @ %s(zBuf) bytes @ </td></tr> @ <tr><th>Number Of Artifacts:</th><td> n = db_int(0, "SELECT count(*) FROM blob"); m = db_int(0, "SELECT count(*) FROM delta"); @ %d(n) (stored as %d(n-m) full text and %d(m) delta blobs) @ </td></tr> if( n>0 ){ int a, b; + Stmt q; @ <tr><th>Uncompressed Artifact Size:</th><td> - t = db_int64(0, "SELECT total(size) FROM blob WHERE size>0"); + db_prepare(&q, "SELECT total(size), avg(size), max(size)" + " FROM blob WHERE size>0"); + db_step(&q); + t = db_column_int64(&q, 0); + szAvg = db_column_int(&q, 1); + szMax = db_column_int(&q, 2); + db_finalize(&q); sqlite3_snprintf(sizeof(zBuf), zBuf, "%lld", t); - @ %d((int)(((double)t)/(double)n)) bytes average, %s(zBuf) bytes total + @ %d(szAvg) bytes average, %d(szMax) bytes max, %s(zBuf) bytes total @ </td></tr> @ <tr><th>Compression Ratio:</th><td> if( t/fsize < 5 ){ b = 10; fsize /= 10; @@ -82,31 +92,34 @@ @ </td></tr> @ <tr><th>Duration Of Project:</th><td> n = db_int(0, "SELECT julianday('now') - (SELECT min(mtime) FROM event)" " + 0.99"); @ %d(n) days + sqlite3_snprintf(sizeof(zBuf), zBuf, "%.2f", n/365.24); + @ or approximately %s(zBuf) years @ </td></tr> @ <tr><th>Project ID:</th><td> @ %h(db_get("project-code","")) @ </td></tr> @ <tr><th>Server ID:</th><td> @ %h(db_get("server-code","")) @ </td></tr> @ <tr><th>Fossil Version:</th><td> - @ %h(MANIFEST_DATE) %h(MANIFEST_VERSION) + @ %h(MANIFEST_DATE) %h(MANIFEST_VERSION) (%h(COMPILER_NAME)) @ </td></tr> @ <tr><th>SQLite Version:</th><td> - @ %h(db_text(0, "SELECT substr(sqlite_source_id(),1,30)")) - @ (%h(SQLITE_VERSION)) + sqlite3_snprintf(sizeof(zBuf), zBuf, "%.19s [%.10s] (%s)", + SQLITE_SOURCE_ID, &SQLITE_SOURCE_ID[20], SQLITE_VERSION); + @ %s(zBuf) @ </td></tr> @ <tr><th>Database Stats:</th><td> @ %d(db_int(0, "PRAGMA %s.page_count", g.zRepoDb)) pages, @ %d(db_int(0, "PRAGMA %s.page_size", g.zRepoDb)) bytes/page, @ %d(db_int(0, "PRAGMA %s.freelist_count", g.zRepoDb)) free pages, @ %s(db_text(0, "PRAGMA %s.encoding", g.zRepoDb)), @ %s(db_text(0, "PRAGMA %s.journal_mode", g.zRepoDb)) mode @ </td></tr> - @ </table></p> + @ </table> style_footer(); } Index: src/style.c ================================================================== --- src/style.c +++ src/style.c @@ -40,10 +40,15 @@ ** Remember that the header has been generated. The footer is omitted ** if an error occurs before the header. */ static int headerHasBeenGenerated = 0; +/* +** remember, if a sidebox was used +*/ +static int sideboxUsed = 0; + /* ** Add a new element to the submenu */ void style_submenu_element( const char *zLabel, @@ -83,11 +88,11 @@ zTitle = vmprintf(zTitleFormat, ap); va_end(ap); cgi_destination(CGI_HEADER); cgi_printf("%s", - "<!DOCTYPE html PUBLIC \"-//W3C/DTD XHTML 1.0 Strict//EN\"" + "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\"" " \"http://www.x3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">"); if( g.thTrace ) Th_Trace("BEGIN_HEADER<br />\n", -1); /* Generate the header up through the main menu */ @@ -96,10 +101,11 @@ Th_Store("baseurl", g.zBaseURL); Th_Store("index_page", db_get("index-page","/home")); Th_Store("current_page", g.zPath); Th_Store("manifest_version", MANIFEST_VERSION); Th_Store("manifest_date", MANIFEST_DATE); + Th_Store("compiler_name", COMPILER_NAME); if( g.zLogin ){ Th_Store("login", g.zLogin); } if( g.thTrace ) Th_Trace("BEGIN_HEADER_SCRIPT<br />\n", -1); Th_Render(zHeader); @@ -106,10 +112,11 @@ if( g.thTrace ) Th_Trace("END_HEADER<br />\n", -1); Th_Unstore("title"); /* Avoid collisions with ticket field names */ cgi_destination(CGI_BODY); g.cgiOutput = 1; headerHasBeenGenerated = 1; + sideboxUsed = 0; } /* ** Draw the footer at the bottom of the page. */ @@ -138,42 +145,46 @@ @ </div> } @ <div class="content"> cgi_destination(CGI_BODY); - /* Put the footer at the bottom of the page. - */ - @ </div><br clear="both"/> + if (sideboxUsed) { + /* Put the footer at the bottom of the page. + ** the additional clear/both is needed to extend the content + ** part to the end of an optional sidebox. + */ + @ <div class="endContent"></div> + } + @ </div> zFooter = db_get("footer", (char*)zDefaultFooter); if( g.thTrace ) Th_Trace("BEGIN_FOOTER<br />\n", -1); Th_Render(zFooter); if( g.thTrace ) Th_Trace("END_FOOTER<br />\n", -1); /* Render trace log if TH1 tracing is enabled. */ if( g.thTrace ){ - cgi_append_content("<font color=\"red\"><hr>\n", -1); + cgi_append_content("<span class=\"thTrace\"><hr />\n", -1); cgi_append_content(blob_str(&g.thLog), blob_size(&g.thLog)); - cgi_append_content("</font>\n", -1); + cgi_append_content("</span>\n", -1); } } /* ** Begin a side-box on the right-hand side of a page. The title and ** the width of the box are given as arguments. The width is usually ** a percentage of total screen width. */ void style_sidebox_begin(const char *zTitle, const char *zWidth){ - @ <table width="%s(zWidth)" align="right" border="1" cellpadding=5 - @ vspace=5 hspace=5> - @ <tr><td> - @ <b>%h(zTitle)</b> + sideboxUsed = 1; + @ <div class="sidebox" style="width:%s(zWidth)"> + @ <div class="sideboxTitle">%h(zTitle)</div> } /* End the side-box */ void style_sidebox_end(void){ - @ </td></tr></table> + @ </div> } /* @-comment: // */ /* ** The default page header. @@ -181,27 +192,27 @@ const char zDefaultHeader[] = @ <html> @ <head> @ <title>$<project_name>: $<title> @ +@ href="$baseurl/timeline.rss" /> @ +@ media="screen" /> @ @ @

$
$</div> -@ <div class="status"><nobr><th1> +@ <div class="title"><small>$<project_name></small><br />$<title></div> +@ <div class="status"><th1> @ if {[info exists login]} { @ puts "Logged in as $login" @ } else { @ puts "Not logged in" @ } -@ </th1></nobr></div> +@ </th1></div> @ </div> @ <div class="mainmenu"><th1> @ html "<a href='$baseurl$index_page'>Home</a> " @ if {[anycap jor]} { @ html "<a href='$baseurl/timeline'>Timeline</a> " @@ -243,10 +254,14 @@ @ </body></html> ; /* ** The default Cascading Style Sheet. +** It's assembled by different strings for each class. +** The default css conatains all definitions. +** The style sheet, send to the client only contains the ones, +** not defined in the user defined css. */ const char zDefaultCSS[] = @ /* General settings for the entire page */ @ body { @ margin: 0ex 1ex; @@ -272,11 +287,11 @@ @ font-weight: bold; @ text-align: center; @ padding: 0 0 0 1em; @ color: #558195; @ vertical-align: bottom; -@ width: 100%; +@ width: 100% ; @ } @ @ /* The login status message in the top right-hand corner */ @ div.status { @ display: table-cell; @@ -284,16 +299,17 @@ @ vertical-align: bottom; @ color: #558195; @ font-size: 0.8em; @ font-weight: bold; @ min-width: 200px; +@ white-space: nowrap; @ } @ @ /* The header across the top of the page */ @ div.header { @ display: table; -@ width: 100%; +@ width: 100% ; @ } @ @ /* The main menu bar that appears at the top of the page beneath @ ** the header */ @ div.mainmenu { @@ -337,10 +353,11 @@ @ padding: 1px 1px 1px 1px; @ font-size: 1.2em; @ font-weight: bold; @ background-color: #558195; @ color: white; +@ white-space: nowrap; @ } @ @ /* The "Date" that occurs on the left hand side of timelines */ @ div.divider { @ background: #a1c4d4; @@ -348,14 +365,16 @@ @ font-size: 1em; font-weight: normal; @ padding: .25em; @ margin: .2em 0 .2em 0; @ float: left; @ clear: left; +@ white-space: nowrap; @ } @ @ /* The footer at the very bottom of the page */ @ div.footer { +@ clear: both; @ font-size: 0.8em; @ margin-top: 12px; @ padding: 5px 10px 5px 10px; @ text-align: right; @ background-color: #558195; @@ -366,11 +385,11 @@ @ div.footer a { color: white; } @ div.footer a:link { color: white; } @ div.footer a:visited { color: white; } @ div.footer a:hover { background-color: white; color: #558195; } @ -@ /* <verbatim> blocks */ +@ /* verbatim blocks */ @ pre.verbatim { @ background-color: #f5f5f5; @ padding: 0.5em; @} @ @@ -378,33 +397,403 @@ @ table.label-value th { @ vertical-align: top; @ text-align: right; @ padding: 0.2ex 2ex; @ } +@ ; + +/* The following table contains bits of default CSS that must +** be included if they are not found in the application-defined +** CSS. +*/ +const struct strctCssDefaults { + char const * const elementClass; /* Name of element needed */ + char const * const comment; /* Comment text */ + char const * const value; /* CSS text */ +} cssDefaultList[] = { + { "", + "", + zDefaultCSS + }, + { "div.sidebox", + "The nomenclature sidebox for branches,..", + @ float: right; + @ background-color: white; + @ border-width: medium; + @ border-style: double; + @ margin: 10; + }, + { "div.sideboxTitle", + "The nomenclature title in sideboxes for branches,..", + @ display: inline; + @ font-weight: bold; + }, + { "div.sideboxDescribed", + "The defined element in sideboxes for branches,..", + @ display: inline; + @ font-weight: bold; + }, + { "span.disabled", + "The defined element in sideboxes for branches,..", + @ color: red; + }, + { "span.timelineDisabled", + "The suppressed duplicates lines in timeline, ..", + @ font-style: italic; + @ font-size: small; + }, + { "table.timelineTable", + "the format for the timeline data table", + @ cellspacing: 0; + @ border: 0; + @ cellpadding: 0 + }, + { "td.timelineTableCell", + "the format for the timeline data cells", + @ valign: top; + @ align: left; + }, + { "span.timelineLeaf", + "the format for the timeline leaf marks", + @ font-weight: bold; + }, + { "a.timelineHistLink", + "the format for the timeline version links", + @ + }, + { "span.timelineHistDsp", + "the format for the timeline version display(no history permission!)", + @ font-weight: bold; + }, + { "td.timelineTime", + "the format for the timeline time display", + @ vertical-align: top; + @ text-align: right; + }, + { "td.timelineGraph", + "the format for the grap placeholder cells in timelines", + @ width: 20; + @ text-align: left; + @ vertical-align: top; + }, + { "a.tagLink", + "the format for the tag links", + @ + }, + { "span.tagDsp", + "the format for the tag display(no history permission!)", + @ font-weight: bold; + }, + { "span.wikiError", + "the format for wiki errors", + @ font-weight: bold; + @ color: red; + }, + { "span.infoTagCancelled", + "the format for fixed/canceled tags,..", + @ font-weight: bold; + @ text-decoration: line-through; + }, + { "span.infoTag", + "the format for tags,..", + @ font-weight: bold; + }, + { "span.wikiTagCancelled", + "the format for fixed/cancelled tags,.. on wiki pages", + @ text-decoration: line-through; + }, + { "table.browser", + "format for the file display table", + @ /* the format for wiki errors */ + @ width: 100% ; + @ border: 0; + }, + { "td.browser", + "format for cells in the file browser", + @ width: 24% ; + @ vertical-align: top; + }, + { "ul.browser", + "format for the list in the file browser", + @ margin-left: 0.5em; + @ padding-left: 0.5em; + }, + { "table.login_out", + "table format for login/out label/input table", + @ text-align: left; + @ margin-right: 10px; + @ margin-left: 10px; + @ margin-top: 10px; + }, + { "div.captcha", + "captcha display options", + @ text-align: center; + }, + { "table.captcha", + "format for the layout table, used for the captcha display", + @ margin: auto; + @ padding: 10px; + @ border-width: 4px; + @ border-style: double; + @ border-color: black; + }, + { "td.login_out_label", + "format for the label cells in the login/out table", + @ text-align: center; + }, + { "span.loginError", + "format for login error messages", + @ color: red; + }, + { "span.note", + "format for leading text for notes", + @ font-weight: bold; + }, + { "span.textareaLabel", + "format for textare labels", + @ font-weight: bold; + }, + { "table.usetupLayoutTable", + "format for the user setup layout table", + @ outline-style: none; + @ padding: 0; + @ margin: 25px; + }, + { "td.usetupColumnLayout", + "format of the columns on the user setup list page", + @ vertical-align: top + }, + { "table.usetupUserList", + "format for the user list table on the user setup page", + @ outline-style: double; + @ outline-width: 1; + @ padding: 10px; + }, + { "th.usetupListUser", + "format for table header user in user list on user setup page", + @ text-align: right; + @ padding-right: 20px; + }, + { "th.usetupListCap", + "format for table header capabilities in user list on user setup page", + @ text-align: center; + @ padding-right: 15px; + }, + { "th.usetupListCon", + "format for table header contact info in user list on user setup page", + @ text-align: left; + }, + { "td.usetupListUser", + "format for table cell user in user list on user setup page", + @ text-align: right; + @ padding-right: 20px; + @ white-space:nowrap; + }, + { "td.usetupListCap", + "format for table cell capabilities in user list on user setup page", + @ text-align: center; + @ padding-right: 15px; + }, + { "td.usetupListCon", + "format for table cell contact info in user list on user setup page", + @ text-align: left + }, + { "div.ueditCapBox", + "layout definition for the capabilities box on the user edit detail page", + @ float: left; + @ margin-right: 20px; + @ margin-bottom: 20px; + }, + { "td.usetupEditLabel", + "format of the label cells in the detailed user edit page", + @ text-align: right; + @ vertical-align: top; + @ white-space: nowrap; + }, + { "span.ueditInheritNobody", + "color for capabilities, inherited by nobody", + @ color: green; + }, + { "span.ueditInheritDeveloper", + "color for capabilities, inherited by developer", + @ color: red; + }, + { "span.ueditInheritReader", + "color for capabilities, inherited by reader", + @ color: black; + }, + { "span.ueditInheritAnonymous", + "color for capabilities, inherited by anonymous", + @ color: blue; + }, + { "span.capability", + "format for capabilites, mentioned on the user edit page", + @ font-weight: bold; + }, + { "span.usertype", + "format for different user types, mentioned on the user edit page", + @ font-weight: bold; + }, + { "span.usertype:before", + "leading text for user types, mentioned on the user edit page", + @ content:"'"; + }, + { "span.usertype:after", + "trailing text for user types, mentioned on the user edit page", + @ content:"'"; + }, + { "p.missingPriv", + "format for missing priviliges note on user setup page", + @ color: blue; + }, + { "span.wikiruleHead", + "format for leading text in wikirules definitions", + @ font-weight: bold; + }, + { "td.tktDspLabel", + "format for labels on ticket display page", + @ text-align: right; + }, + { "td.tktDspValue", + "format for values on ticket display page", + @ text-align: left; + @ vertical-align: top; + @ background-color: #d0d0d0; + }, + { "span.tktError", + "format for ticket error messages", + @ color: red; + @ font-weight: bold; + }, + { "table.rpteditex", + "format for example tables on the report edit page", + @ float: right; + @ margin: 0; + @ padding: 0; + @ width: 125px; + @ text-align: center; + @ border-collapse: collapse; + @ border-spacing: 0; + }, + { "td.rpteditex", + "format for example table cells on the report edit page", + @ border-width: thin; + @ border-color: #000000; + @ border-style: solid; + }, + { "input.checkinUserColor", + "format for user color input on checkin edit page", + @ # no special definitions, class defined, to enable color pickers, f.e.: + @ # add the color picker found at http:jscolor.com as java script include + @ # to the header and configure the java script file with + @ # 1. use as bindClass :checkinUserColor + @ # 2. change the default hash adding behaviour to ON + @ # or change the class defition of element identified by id="clrcust" + @ # to a standard jscolor definition with java script in the footer. + }, + { "div.endContent", + "format for end of content area, to be used to clear page flow(sidebox on branch,..", + @ clear: both; + }, + { "p.generalError", + "format for general errors", + @ color: red; + }, + { "p.tktsetupError", + "format for tktsetup errors", + @ color: red; + @ font-weight: bold; + }, + { "p.thmainError", + "format for th script errors", + @ color: red; + @ font-weight: bold; + }, + { "span.thTrace", + "format for th script trace messages", + @ color: red; + }, + { "p:reportError", + "format for report configuration errors", + @ color: red; + @ font-weight: bold; + }, + { "blockquote.reportError", + "format for report configuration errors", + @ color: red; + @ font-weight: bold; + }, + { "p.noMoreShun", + "format for artifact lines, no longer shunned", + @ color: blue; + }, + { "p.shunned", + "format for artifact lines beeing shunned", + @ color: blue; + }, + { 0, + 0, + 0 + } +}; + +/* +** Append all of the default CSS to the CGI output. +*/ +void cgi_append_default_css(void) { + int i; + + for (i=0;cssDefaultList[i].elementClass;i++){ + if (cssDefaultList[i].elementClass[0]){ + cgi_printf("/* %s */\n%s {\n%s\n}\n\n", + cssDefaultList[i].comment, + cssDefaultList[i].elementClass, + cssDefaultList[i].value + ); + }else{ + cgi_printf("%s", + cssDefaultList[i].value + ); + } + } +} + /* ** WEBPAGE: style.css */ void page_style_css(void){ - char *zCSS = 0; + const char *zCSS = 0; + int i; cgi_set_content_type("text/css"); zCSS = db_get("css",(char*)zDefaultCSS); + /* append user defined css */ cgi_append_content(zCSS, -1); + /* add special missing definitions */ + for (i=1;cssDefaultList[i].elementClass;i++) + if (!strstr(zCSS,cssDefaultList[i].elementClass)) { + cgi_append_content("/* ", -1); + cgi_append_content(cssDefaultList[i].comment, -1); + cgi_append_content(" */\n", -1); + cgi_append_content(cssDefaultList[i].elementClass, -1); + cgi_append_content(" {\n", -1); + cgi_append_content(cssDefaultList[i].value, -1); + cgi_append_content("}\n\n", -1); + } g.isConst = 1; } /* ** WEBPAGE: test_env */ void page_test_env(void){ style_header("Environment Test"); -#if !defined(__MINGW32__) - @ uid=%d(getuid()), gid=%d(getgid())<br> +#if !defined(_WIN32) + @ uid=%d(getuid()), gid=%d(getgid())<br /> #endif - @ g.zBaseURL = %h(g.zBaseURL)<br> - @ g.zTop = %h(g.zTop)<br> - @ g.zRepositoryName = %h(g.zRepositoryName)<br> + @ g.zBaseURL = %h(g.zBaseURL)<br /> + @ g.zTop = %h(g.zTop)<br /> cgi_print_all(); style_footer(); } Index: src/sync.c ================================================================== --- src/sync.c +++ src/sync.c @@ -56,11 +56,11 @@ } zUrl = db_get("last-sync-url", 0); if( zUrl==0 ){ return; /* No default server */ } - zPw = db_get("last-sync-pw", 0); + zPw = unobscure(db_get("last-sync-pw", 0)); url_parse(zUrl); if( g.urlUser!=0 && g.urlPasswd==0 ){ g.urlPasswd = mprintf("%s", zPw); } if( (flags & AUTOSYNC_PULL)!=0 && db_get_boolean("auto-shun",1) ){ @@ -93,11 +93,11 @@ url_proxy_options(); db_find_and_open_repository(1); db_open_config(0); if( g.argc==2 ){ zUrl = db_get("last-sync-url", 0); - zPw = db_get("last-sync-pw", 0); + zPw = unobscure(db_get("last-sync-pw", 0)); if( db_get_boolean("auto-sync",1) ) configSync = CONFIGSET_SHUN; }else if( g.argc==3 ){ zUrl = g.argv[2]; } if( zUrl==0 ){ @@ -105,11 +105,11 @@ usage("URL"); } url_parse(zUrl); if( !g.dontKeepUrl ){ db_set("last-sync-url", g.urlCanonical, 0); - if( g.urlPasswd ) db_set("last-sync-pw", g.urlPasswd, 0); + if( g.urlPasswd ) db_set("last-sync-pw", obscure(g.urlPasswd), 0); } if( g.urlUser!=0 && g.urlPasswd==0 ){ if( zPw==0 ){ url_prompt_for_password(); }else{ @@ -232,11 +232,11 @@ if( g.urlUser && g.urlPasswd==0 ){ url_prompt_for_password(); } db_set("last-sync-url", g.urlCanonical, 0); if( g.urlPasswd ){ - db_set("last-sync-pw", g.urlPasswd, 0); + db_set("last-sync-pw", obscure(g.urlPasswd), 0); }else{ db_unset("last-sync-pw", 0); } } } Index: src/tag.c ================================================================== --- src/tag.c +++ src/tag.c @@ -258,11 +258,13 @@ void tag_add_artifact( const char *zPrefix, /* Prefix to prepend to tag name */ const char *zTagname, /* The tag to add or cancel */ const char *zObjName, /* Name of object attached to */ const char *zValue, /* Value for the tag. Might be NULL */ - int tagtype /* 0:cancel 1:singleton 2:propagated */ + int tagtype, /* 0:cancel 1:singleton 2:propagated */ + const char *zDateOvrd, /* Override date string */ + const char *zUserOvrd /* Override user name */ ){ int rid; int nrid; char *zDate; Blob uuid; @@ -289,21 +291,21 @@ " a hexadecimal artifact ID", zTagname ); } #endif - zDate = db_text(0, "SELECT datetime('now')"); + zDate = date_in_standard_format(zDateOvrd ? zDateOvrd : "now"); zDate[10] = 'T'; blob_appendf(&ctrl, "D %s\n", zDate); blob_appendf(&ctrl, "T %c%s%F %b", zTagtype[tagtype], zPrefix, zTagname, &uuid); if( tagtype>0 && zValue && zValue[0] ){ blob_appendf(&ctrl, " %F\n", zValue); }else{ blob_appendf(&ctrl, "\n"); } - blob_appendf(&ctrl, "U %F\n", g.zLogin); + blob_appendf(&ctrl, "U %F\n", zUserOvrd ? zUserOvrd : g.zLogin); md5sum_blob(&ctrl, &cksum); blob_appendf(&ctrl, "Z %b\n", &cksum); nrid = content_put(&ctrl, 0, 0); manifest_crosslink(nrid, &ctrl); } @@ -352,10 +354,14 @@ ** ** fossil update tag:decaf ** ** will assume that "decaf" is a tag/branch name. ** +** only allow --date-override and --user-override in +** %fossil tag add --date-override 'YYYY-MMM-DD HH:MM:SS' \\ +** --user-override user +** in order to import history from other scm systems */ void tag_cmd(void){ int n; int fRaw = find_option("raw","",0)!=0; int fPropagate = find_option("propagate","",0)!=0; @@ -370,16 +376,19 @@ goto tag_cmd_usage; } if( strncmp(g.argv[2],"add",n)==0 ){ char *zValue; + const char *zDateOvrd = find_option("date-override",0,1); + const char *zUserOvrd = find_option("user-override",0,1); if( g.argc!=5 && g.argc!=6 ){ usage("add ?--raw? ?--propagate? TAGNAME CHECK-IN ?VALUE?"); } zValue = g.argc==6 ? g.argv[5] : 0; db_begin_transaction(); - tag_add_artifact(zPrefix, g.argv[3], g.argv[4], zValue, 1+fPropagate); + tag_add_artifact(zPrefix, g.argv[3], g.argv[4], zValue, + 1+fPropagate,zDateOvrd,zUserOvrd); db_end_transaction(0); }else if( strncmp(g.argv[2],"branch",n)==0 ){ fossil_fatal("the \"fossil tag branch\" command is discontinued\n" @@ -389,11 +398,11 @@ if( strncmp(g.argv[2],"cancel",n)==0 ){ if( g.argc!=5 ){ usage("cancel ?--raw? TAGNAME CHECK-IN"); } db_begin_transaction(); - tag_add_artifact(zPrefix, g.argv[3], g.argv[4], 0, 0); + tag_add_artifact(zPrefix, g.argv[3], g.argv[4], 0, 0, 0, 0); db_end_transaction(0); }else if( strncmp(g.argv[2],"find",n)==0 ){ Stmt q; @@ -514,13 +523,14 @@ ); @ <ul> while( db_step(&q)==SQLITE_ROW ){ const char *zName = db_column_text(&q, 0); if( g.okHistory ){ - @ <li><a href=%s(g.zBaseURL)/timeline?t=%T(zName)>%h(zName)</a></li> + @ <li><a class="tagLink" href="%s(g.zBaseURL)/timeline?t=%T(zName)"> + @ %h(zName)</a></li> }else{ - @ <li><strong>%h(zName)</strong></li> + @ <li><span class="tagDsp">%h(zName)</span></li> } } @ </ul> db_finalize(&q); style_footer(); @@ -542,13 +552,14 @@ rid ); while( db_step(&q)==SQLITE_ROW ){ const char *zTagName = db_column_text(&q, 0); if( g.okHistory ){ - @ <a href="%s(g.zBaseURL)/timeline?t=%T(zTagName)">[%h(zTagName)]</a> + @ <a class="tagLink" href="%s(g.zBaseURL)/timeline?t=%T(zTagName)"> + @ [%h(zTagName)]</a> }else{ - @ <b>[%h(zTagName)]</b> + @ <span class="tagDsp">[%h(zTagName)]</span> } } db_finalize(&q); } @@ -573,14 +584,14 @@ " ORDER BY event.mtime DESC", timeline_query_for_www() ); www_print_timeline(&q, 0, tagtimeline_extra); db_finalize(&q); - @ <br clear="both"> - @ <script> + @ <br /> + @ <script type="text/JavaScript"> @ function xin(id){ @ } @ function xout(id){ @ } @ </script> style_footer(); } Index: src/th.c ================================================================== --- src/th.c +++ src/th.c @@ -846,11 +846,11 @@ /* Gobble up input a word at a time until the end of the command ** (a semi-colon or end of line). */ while( rc==TH_OK && *zInput!=';' && !thEndOfLine(zInput, nInput) ){ - int nWord; + int nWord=0; thNextSpace(interp, zInput, nInput, &nSpace); rc = thNextWord(interp, &zInput[nSpace], nInput-nSpace, &nWord, 1); zInput += (nSpace+nWord); nInput -= (nSpace+nWord); } Index: src/th_lang.c ================================================================== --- src/th_lang.c +++ src/th_lang.c @@ -7,10 +7,11 @@ ** declared in th.h, so this file serves as both a part of the language ** implementation and an example of how to extend the language with ** new commands. */ +#include "config.h" #include "th.h" #include <string.h> #include <assert.h> int Th_WrongNumArgs(Th_Interp *interp, const char *zMsg){ @@ -571,11 +572,11 @@ return Th_WrongNumArgs(interp, "return ?value?"); } if( argc==2 ){ Th_SetResult(interp, argv[1], argl[1]); } - return (int)ctx; + return FOSSIL_PTR_TO_INT(ctx); } /* ** TH Syntax: ** Index: src/th_main.c ================================================================== --- src/th_main.c +++ src/th_main.c @@ -30,11 +30,11 @@ /* ** Implementations of malloc() and free() to pass to the interpreter. */ static void *xMalloc(unsigned int n){ - void *p = malloc(n); + void *p = fossil_malloc(n); if( p ){ nOutstandingMalloc++; } return p; } @@ -278,11 +278,12 @@ blob_reset(&name); for(i=0; i<nElem; i++){ zH = htmlize((char*)azElem[i], aszElem[i]); if( zValue && aszElem[i]==nValue && memcmp(zValue, azElem[i], nValue)==0 ){ - z = mprintf("<option value=\"%s\" selected>%s</option>", zH, zH); + z = mprintf("<option value=\"%s\" selected=\"selected\">%s</option>", + zH, zH); }else{ z = mprintf("<option value=\"%s\">%s</option>", zH, zH); } free(zH); sendText(z, -1, 0); @@ -434,20 +435,20 @@ int inBracket = 0; if( z[0]=='<' ){ inBracket = 1; z++; } - if( z[0]==':' && z[1]==':' && isalpha(z[2]) ){ + if( z[0]==':' && z[1]==':' && fossil_isalpha(z[2]) ){ z += 3; i += 3; - }else if( isalpha(z[0]) ){ + }else if( fossil_isalpha(z[0]) ){ z ++; i += 1; }else{ return 0; } - while( isalnum(z[0]) || z[0]=='_' ){ + while( fossil_isalnum(z[0]) || z[0]=='_' ){ z++; i++; } if( inBracket ){ if( z[0]!='>' ) return 0; @@ -503,14 +504,14 @@ }else{ i++; } } if( rc==TH_ERROR ){ - sendText("<hr><p><font color=\"red\"><b>ERROR: ", -1, 0); + sendText("<hr><p class=\"thmainError\">ERROR: ", -1, 0); zResult = (char*)Th_GetResult(g.interp, &n); sendText((char*)zResult, n, 1); - sendText("</b></font></p>", -1, 0); + sendText("</p>", -1, 0); }else{ sendText(z, i, 0); } return rc; } Index: src/timeline.c ================================================================== --- src/timeline.c +++ src/timeline.c @@ -48,13 +48,14 @@ */ void hyperlink_to_uuid(const char *zUuid){ char zShortUuid[UUID_SIZE+1]; shorten_uuid(zShortUuid, zUuid); if( g.okHistory ){ - @ <a href="%s(g.zBaseURL)/info/%s(zShortUuid)">[%s(zShortUuid)]</a> + @ <a class="timelineHistLink" href="%s(g.zBaseURL)/info/%s(zShortUuid)"> + @ [%s(zShortUuid)]</a> }else{ - @ <b>[%s(zShortUuid)]</b> + @ <span class="timelineHistDsp">[%s(zShortUuid)]</span> } } /* ** Generate a hyperlink that invokes javascript to highlight @@ -83,11 +84,11 @@ void hyperlink_to_diff(const char *zV1, const char *zV2){ if( g.okHistory ){ if( zV2==0 ){ @ <a href="%s(g.zBaseURL)/diff?v2=%s(zV1)">[diff]</a> }else{ - @ <a href="%s(g.zBaseURL)/diff?v1=%s(zV1)&v2=%s(zV2)">[diff]</a> + @ <a href="%s(g.zBaseURL)/diff?v1=%s(zV1)&v2=%s(zV2)">[diff]</a> } } } /* @@ -109,11 +110,11 @@ */ void hyperlink_to_user(const char *zU, const char *zD, const char *zSuf){ if( zSuf==0 ) zSuf = ""; if( g.okHistory ){ if( zD && zD[0] ){ - @ <a href="%s(g.zTop)/timeline?c=%T(zD)&u=%T(zU)">%h(zU)</a>%s(zSuf) + @ <a href="%s(g.zTop)/timeline?c=%T(zD)&u=%T(zU)">%h(zU)</a>%s(zSuf) }else{ @ <a href="%s(g.zTop)/timeline?u=%T(zU)">%h(zU)</a>%s(zSuf) } }else{ @ %s(zU) @@ -164,11 +165,11 @@ ** 4. User ** 5. True if is a leaf ** 6. background color ** 7. type ("ci", "w", "t") ** 8. list of symbolic tags. -** 9. tagid for ticket or wiki +** 9. tagid for ticket or wiki or event ** 10. Short comment to user for repeated tickets and wiki */ void www_print_timeline( Stmt *pQuery, /* Query to implement the timeline */ int tmFlags, /* Flags controlling display behavior */ @@ -189,14 +190,17 @@ }else{ wikiFlags = WIKI_INLINE | WIKI_NOBLOCK; } if( tmFlags & TIMELINE_GRAPH ){ pGraph = graph_init(); + /* style is not moved to css, because this is + ** a technical div for the timeline graph + */ @ <div id="canvas" style="position:relative;width:1px;height:1px;"></div> } - @ <table cellspacing=0 border=0 cellpadding=0> + @ <table class="timelineTable"> blob_zero(&comment); while( db_step(pQuery)==SQLITE_ROW ){ int rid = db_column_int(pQuery, 0); const char *zUuid = db_column_text(pQuery, 1); int isLeaf = db_column_int(pQuery, 5); @@ -218,30 +222,30 @@ } } } prevTagid = tagid; if( suppressCnt ){ - @ <tr><td><td><td> - @ <small><i>... %d(suppressCnt) similar - @ event%s(suppressCnt>1?"s":"") omitted.</i></small></tr> + @ <tr><td /><td /><td> + @ <span class="timelineDisabled">... %d(suppressCnt) similar + @ event%s(suppressCnt>1?"s":"") omitted.</span></td></tr> suppressCnt = 0; } if( strcmp(zType,"div")==0 ){ - @ <tr><td colspan=3><hr></td></tr> + @ <tr><td colspan="3"><hr /></td></tr> continue; } if( memcmp(zDate, zPrevDate, 10) ){ sprintf(zPrevDate, "%.10s", zDate); @ <tr><td> - @ <div class="divider"><nobr>%s(zPrevDate)</nobr></div> + @ <div class="divider">%s(zPrevDate)</div> @ </td></tr> } memcpy(zTime, &zDate[11], 5); zTime[5] = 0; @ <tr> - @ <td valign="top" align="right">%s(zTime)</td> - @ <td width="20" align="left" valign="top"> + @ <td class="timelineTime">%s(zTime)</td> + @ <td class="timelineGraph"> if( pGraph && zType[0]=='c' ){ int nParent = 0; int aParent[32]; const char *zBr; int gidx; @@ -267,26 +271,29 @@ } gidx = graph_add_row(pGraph, rid, nParent, aParent, zBr, zBgClr); db_reset(&qbranch); @ <div id="m%d(gidx)"></div> } + @</td> if( zBgClr && zBgClr[0] ){ - @ <td valign="top" align="left" bgcolor="%h(zBgClr)"> + @ <td class="timelineTableCell" style="background-color: %h(zBgClr);"> }else{ - @ <td valign="top" align="left"> + @ <td class="timelineTableCell"> } if( zType[0]=='c' ){ hyperlink_to_uuid(zUuid); if( isLeaf ){ if( db_exists("SELECT 1 FROM tagxref" " WHERE rid=%d AND tagid=%d AND tagtype>0", rid, TAG_CLOSED) ){ - @ <b>Closed-Leaf:</b> + @ <span class="timelineLeaf">Closed-Leaf:</span> }else{ - @ <b>Leaf:</b> + @ <span class="timelineLeaf">Leaf:</span> } } + }else if( zType[0]=='e' && tagid ){ + hyperlink_to_event_tagid(tagid); }else if( (tmFlags & TIMELINE_ARTID)!=0 ){ hyperlink_to_uuid(zUuid); } db_column_blob(pQuery, commentColumn, &comment); if( mxWikiLen>0 && blob_size(&comment)>mxWikiLen ){ @@ -309,22 +316,27 @@ xExtra(rid); } @ </td></tr> } if( suppressCnt ){ - @ <tr><td><td><td> - @ <small><i>... %d(suppressCnt) similar - @ event%s(suppressCnt>1?"s":"") omitted.</i></small></tr> + @ <tr><td /><td /><td> + @ <span class="timelineDisabled">... %d(suppressCnt) similar + @ event%s(suppressCnt>1?"s":"") omitted.</span></td></tr> suppressCnt = 0; } if( pGraph ){ graph_finish(pGraph, (tmFlags & TIMELINE_DISJOINT)!=0); if( pGraph->nErr ){ graph_free(pGraph); pGraph = 0; }else{ - @ <tr><td><td><div style="width:%d(pGraph->mxRail*20+30)px;"></div> + /* style is not moved to css, because this is + ** a technical div for the timeline graph + */ + @ <tr><td /><td> + @ <div id="grbtm" style="width:%d(pGraph->mxRail*20+30)px;"></div> + @ </td></tr> } } @ </table> timeline_output_graph_javascript(pGraph); } @@ -336,11 +348,12 @@ void timeline_output_graph_javascript(GraphContext *pGraph){ if( pGraph && pGraph->nErr==0 ){ GraphRow *pRow; int i; char cSep; - @ <script type="text/JavaScript"> + @ <script type="text/JavaScript"> + @ /* <![CDATA[ */ cgi_printf("var rowinfo = [\n"); for(pRow=pGraph->pFirst; pRow; pRow=pRow->pNext){ cgi_printf("{id:\"m%d\",bg:\"%s\",r:%d,d:%d,mo:%d,mu:%d,u:%d,au:", pRow->idx, pRow->zBgClr, @@ -352,20 +365,20 @@ ); cSep = '['; for(i=0; i<GR_MAX_RAIL; i++){ if( i==pRow->iRail ) continue; if( pRow->aiRaiser[i]>0 ){ - cgi_printf("%c%d,%d", cSep, pGraph->railMap[i], pRow->aiRaiser[i]); + cgi_printf("%c%d,%d", cSep, i, pRow->aiRaiser[i]); cSep = ','; } } if( cSep=='[' ) cgi_printf("["); cgi_printf("],mi:"); cSep = '['; for(i=0; i<GR_MAX_RAIL; i++){ if( pRow->mergeIn & (1<<i) ){ - cgi_printf("%c%d", cSep, pGraph->railMap[i]); + cgi_printf("%c%d", cSep, i); cSep = ','; } } if( cSep=='[' ) cgi_printf("["); cgi_printf("]}%s", pRow->pNext ? ",\n" : "];\n"); @@ -482,15 +495,15 @@ @ var width = nrail*20; @ for(var i in rowinfo){ @ rowinfo[i].y = absoluteY(rowinfo[i].id) + 10 - canvasY; @ rowinfo[i].x = left + rowinfo[i].r*20; @ } - @ var btm = rowinfo[rowinfo.length-1].y + 20; + @ var btm = absoluteY("grbtm") + 10 - canvasY; @ if( btm<32768 ){ @ canvasDiv.innerHTML = '<canvas id="timeline-canvas" '+ @ 'style="position:absolute;left:'+(left-5)+'px;"' + - @ ' width="'+width+'" height="'+btm+'"></canvas>'; + @ ' width="'+width+'" height="'+btm+'"><'+'/canvas>'; @ realCanvas = document.getElementById('timeline-canvas'); @ }else{ @ realCanvas = 0; @ } @ var context; @@ -518,10 +531,11 @@ @ lastY = h; @ } @ setTimeout("checkHeight();", 1000); @ } @ checkHeight(); + @ /* ]]> */ @ </script> } } /* @@ -627,11 +641,11 @@ ** p=RID artifact RID and up to COUNT parents and ancestors ** d=RID artifact RID and up to COUNT descendants ** t=TAGID show only check-ins with the given tagid ** r=TAGID show check-ins related to tagid ** u=USER only if belonging to this user -** y=TYPE 'ci', 'w', 't' +** y=TYPE 'ci', 'w', 't', 'e' ** s=TEXT string search (comment and brief) ** ng Suppress the graph if present ** ** p= and d= can appear individually or together. If either p= or d= ** appear, then u=, y=, a=, and b= are ignored. @@ -737,22 +751,22 @@ }else{ blob_appendf(&desc, " of check-in [%.10s]", zUuid); } }else{ int n; - const char *zEType = "event"; + const char *zEType = "timeline item"; char *zDate; char *zNEntry = mprintf("%d", nEntry); url_initialize(&url, "timeline"); url_add_parameter(&url, "n", zNEntry); if( tagid>0 ){ - zType = "ci"; + if( zType[0]!='e' ) zType = "ci"; blob_appendf(&sql, "AND (EXISTS(SELECT 1 FROM tagxref" " WHERE tagid=%d AND tagtype>0 AND rid=blob.rid)", tagid); - if( zBrName ){ + if( zBrName && zType[0]=='c' ){ /* The next two blob_appendf() calls add SQL that causes checkins that ** are not part of the branch which are parents or childen of the branch ** to be included in the report. This related check-ins are useful ** in helping to visualize what has happened on a quiescent branch ** that is infrequently merged with a much more activate branch. @@ -769,10 +783,11 @@ } blob_appendf(&sql, ")"); } if( (zType[0]=='w' && !g.okRdWiki) || (zType[0]=='t' && !g.okRdTkt) + || (zType[0]=='e' && !g.okRdWiki) || (zType[0]=='c' && !g.okRead) ){ zType = "all"; } if( zType[0]=='a' ){ @@ -782,11 +797,11 @@ if( g.okRead ){ blob_appendf(&sql, "%c'ci'", cSep); cSep = ','; } if( g.okRdWiki ){ - blob_appendf(&sql, "%c'w'", cSep); + blob_appendf(&sql, "%c'w','e'", cSep); cSep = ','; } if( g.okRdTkt ){ blob_appendf(&sql, "%c't'", cSep); cSep = ','; @@ -800,10 +815,12 @@ zEType = "checkin"; }else if( zType[0]=='w' ){ zEType = "wiki edit"; }else if( zType[0]=='t' ){ zEType = "ticket change"; + }else if( zType[0]=='e' ){ + zEType = "event"; } } if( zUser ){ blob_appendf(&sql, " AND event.user=%Q", zUser); url_add_parameter(&url, "u", zUser); @@ -813,11 +830,11 @@ " AND (event.comment LIKE '%%%q%%' OR event.brief LIKE '%%%q%%')", zSearch, zSearch); url_add_parameter(&url, "s", zSearch); } if( zAfter ){ - while( isspace(zAfter[0]) ){ zAfter++; } + while( fossil_isspace(zAfter[0]) ){ zAfter++; } if( zAfter[0] ){ blob_appendf(&sql, " AND event.mtime>=(SELECT julianday(%Q, 'utc'))" " ORDER BY event.mtime ASC", zAfter); url_add_parameter(&url, "a", zAfter); @@ -824,21 +841,21 @@ zBefore = 0; }else{ zAfter = 0; } }else if( zBefore ){ - while( isspace(zBefore[0]) ){ zBefore++; } + while( fossil_isspace(zBefore[0]) ){ zBefore++; } if( zBefore[0] ){ blob_appendf(&sql, " AND event.mtime<=(SELECT julianday(%Q, 'utc'))" " ORDER BY event.mtime DESC", zBefore); url_add_parameter(&url, "b", zBefore); }else{ zBefore = 0; } }else if( zCirca ){ - while( isspace(zCirca[0]) ){ zCirca++; } + while( fossil_isspace(zCirca[0]) ){ zCirca++; } if( zCirca[0] ){ double rCirca = db_double(0.0, "SELECT julianday(%Q, 'utc')", zCirca); Blob sql2; blob_init(&sql2, blob_str(&sql), -1); blob_appendf(&sql2, @@ -882,15 +899,15 @@ }else if( zBrName ){ blob_appendf(&desc, " related to \"%h\"", zBrName); tmFlags |= TIMELINE_DISJOINT; } if( zAfter ){ - blob_appendf(&desc, " occurring on or after %h.<br>", zAfter); + blob_appendf(&desc, " occurring on or after %h.<br />", zAfter); }else if( zBefore ){ - blob_appendf(&desc, " occurring on or before %h.<br>", zBefore); + blob_appendf(&desc, " occurring on or before %h.<br />", zBefore); }else if( zCirca ){ - blob_appendf(&desc, " occurring around %h.<br>", zCirca); + blob_appendf(&desc, " occurring around %h.<br />", zCirca); } if( zSearch ){ blob_appendf(&desc, " matching \"%h\"", zSearch); } if( g.okHistory ){ @@ -914,19 +931,25 @@ timeline_submenu(&url, "Checkins Only", "y", "ci", 0); } if( zType[0]!='t' && g.okRdTkt ){ timeline_submenu(&url, "Tickets Only", "y", "t", 0); } + if( zType[0]!='e' && g.okRdWiki ){ + timeline_submenu(&url, "Events Only", "y", "e", 0); + } } if( nEntry>20 ){ - timeline_submenu(&url, "20 Events", "n", "20", 0); + timeline_submenu(&url, "20 Entries", "n", "20", 0); } if( nEntry<200 ){ - timeline_submenu(&url, "200 Events", "n", "200", 0); + timeline_submenu(&url, "200 Entries", "n", "200", 0); } } } + if( P("showsql") ){ + @ <blockquote>%h(blob_str(&sql))</blockquote> + } blob_zero(&sql); db_prepare(&q, "SELECT * FROM timeline ORDER BY timestamp DESC /*scan*/"); @ <h2>%b(&desc)</h2> blob_reset(&desc); www_print_timeline(&q, tmFlags, 0); @@ -1037,12 +1060,12 @@ */ static int isIsoDate(const char *z){ return strlen(z)==10 && z[4]=='-' && z[7]=='-' - && isdigit(z[0]) - && isdigit(z[5]); + && fossil_isdigit(z[0]) + && fossil_isdigit(z[5]); } /* ** COMMAND: timeline ** Index: src/tkt.c ================================================================== --- src/tkt.c +++ src/tkt.c @@ -53,14 +53,11 @@ db_prepare(&q, "PRAGMA table_info(ticket)"); while( db_step(&q)==SQLITE_ROW ){ const char *zField = db_column_text(&q, 1); if( strncmp(zField,"tkt_",4)==0 ) continue; if( nField%10==0 ){ - azField = realloc(azField, sizeof(azField)*3*(nField+10) ); - if( azField==0 ){ - fossil_fatal("out of memory"); - } + azField = fossil_realloc(azField, sizeof(azField)*3*(nField+10) ); } azField[nField] = mprintf("%s", zField); nField++; } db_finalize(&q); @@ -217,25 +214,25 @@ */ void ticket_rebuild_entry(const char *zTktUuid){ char *zTag = mprintf("tkt-%s", zTktUuid); int tagid = tag_findid(zTag, 1); Stmt q; - Manifest manifest; - Blob content; + Manifest *pTicket; int createFlag = 1; db_multi_exec( "DELETE FROM ticket WHERE tkt_uuid=%Q", zTktUuid ); db_prepare(&q, "SELECT rid FROM tagxref WHERE tagid=%d ORDER BY mtime",tagid); while( db_step(&q)==SQLITE_ROW ){ int rid = db_column_int(&q, 0); - content_get(rid, &content); - manifest_parse(&manifest, &content); - ticket_insert(&manifest, createFlag, rid); - manifest_ticket_event(rid, &manifest, createFlag, tagid); - manifest_clear(&manifest); + pTicket = manifest_get(rid, CFTYPE_TICKET); + if( pTicket ){ + ticket_insert(pTicket, createFlag, rid); + manifest_ticket_event(rid, pTicket, createFlag, tagid); + manifest_destroy(pTicket); + } createFlag = 0; } db_finalize(&q); } @@ -313,11 +310,11 @@ style_submenu_element("New Ticket", "Create a new ticket", "%s/tktnew", g.zTop); } if( g.okApndTkt && g.okAttach ){ style_submenu_element("Attach", "Add An Attachment", - "%s/attachadd?tkt=%T&from=%s/tktview/%t", + "%s/attachadd?tkt=%T&from=%s/tktview/%t", g.zTop, zUuid, g.zTop, zUuid); } style_header("View Ticket"); if( g.thTrace ) Th_Trace("BEGIN_TKTVIEW<br />\n", -1); ticket_init(); @@ -342,20 +339,27 @@ while( db_step(&q)==SQLITE_ROW ){ const char *zDate = db_column_text(&q, 0); const char *zFile = db_column_text(&q, 1); const char *zUser = db_column_text(&q, 2); if( cnt==0 ){ - @ <hr><h2>Attachments:</h2> + @ <hr /><h2>Attachments:</h2> @ <ul> } cnt++; - @ <li><a href="%s(g.zTop)/attachview?tkt=%s(zFullName)&file=%t(zFile)"> - @ %h(zFile)</a> add by %h(zUser) on + @ <li> + if( g.okRead && g.okHistory ){ + @ <a href="%s(g.zTop)/attachview?tkt=%s(zFullName)&file=%t(zFile)"> + @ %h(zFile)</a> + }else{ + @ %h(zFile) + } + @ added by %h(zUser) on hyperlink_to_date(zDate, "."); if( g.okWrTkt && g.okAttach ){ - @ [<a href="%s(g.zTop)/attachdelete?tkt=%s(zFullName)&file=%t(zFile)&from=%s(g.zTop)/tktview%%3fname=%s(zFullName)">delete</a>] + @ [<a href="%s(g.zTop)/attachdelete?tkt=%s(zFullName)&file=%t(zFile)&from=%s(g.zTop)/tktview%%3fname=%s(zFullName)">delete</a>] } + @ </li> } if( cnt ){ @ </ul> } db_finalize(&q); @@ -438,11 +442,11 @@ blob_appendf(&tktchng, "J +%s %z\n", azField[i], fossilize(azAppend[i], -1)); }else{ zValue = Th_Fetch(azField[i], &nValue); if( zValue ){ - while( nValue>0 && isspace(zValue[nValue-1]) ){ nValue--; } + while( nValue>0 && fossil_isspace(zValue[nValue-1]) ){ nValue--; } if( strncmp(zValue, azValue[i], nValue) || strlen(azValue[i])!=nValue ){ if( strncmp(azField[i], "private_", 8)==0 ){ zValue = db_conceal(zValue, nValue); blob_appendf(&tktchng, "J %s %s\n", azField[i], zValue); }else{ @@ -462,11 +466,18 @@ *(const char**)pUuid = zUuid; blob_appendf(&tktchng, "K %s\n", zUuid); blob_appendf(&tktchng, "U %F\n", g.zLogin ? g.zLogin : ""); md5sum_blob(&tktchng, &cksum); blob_appendf(&tktchng, "Z %b\n", &cksum); - if( g.thTrace ){ + if( g.zPath[0]=='d' ){ + /* If called from /debug_tktnew or /debug_tktedit... */ + @ <font color="blue"> + @ <p>Ticket artifact that would have been submitted:</p> + @ <blockquote><pre>%h(blob_str(&tktchng))</pre></blockquote> + @ <hr /></font> + return TH_OK; + }else if( g.thTrace ){ Th_Trace("submit_ticket {\n<blockquote><pre>\n%h\n</pre></blockquote>\n" "}<br />\n", blob_str(&tktchng)); }else{ rid = content_put(&tktchng, 0, 0); @@ -506,12 +517,13 @@ if( g.thTrace ) Th_Trace("BEGIN_TKTNEW<br />\n", -1); ticket_init(); getAllTicketFields(); initializeVariablesFromDb(); initializeVariablesFromCGI(); - @ <form method="POST" action="%s(g.zBaseURL)/%s(g.zPath)"> + @ <form method="post" action="%s(g.zBaseURL)/%s(g.zPath)"><p> login_insert_csrf_secret(); + @ </p> zScript = ticket_newpage_code(); Th_Store("login", g.zLogin); Th_Store("date", db_text(0, "SELECT datetime('now')")); Th_CreateCommand(g.interp, "submit_ticket", submitTicketCmd, (void*)&zNewUuid, 0); @@ -549,34 +561,36 @@ cgi_redirectf("tktview?name=%T", zName); } style_header("Edit Ticket"); if( zName==0 || (nName = strlen(zName))<4 || nName>UUID_SIZE || !validate16(zName,nName) ){ - @ <font color="red"><b>Not a valid ticket id: \"%h(zName)\"</b></font> + @ <span class="tktError">Not a valid ticket id: \"%h(zName)\"</span> style_footer(); return; } nRec = db_int(0, "SELECT count(*) FROM ticket WHERE tkt_uuid GLOB '%q*'", zName); if( nRec==0 ){ - @ <font color="red"><b>No such ticket: \"%h(zName)\"</b></font> + @ <span class="tktError">No such ticket: \"%h(zName)\"</span> style_footer(); return; } if( nRec>1 ){ - @ <font color="red"><b>%d(nRec) tickets begin with: \"%h(zName)\"</b></font> + @ <span class="tktError">%d(nRec) tickets begin with: + @ \"%h(zName)\"</span> style_footer(); return; } if( g.thTrace ) Th_Trace("BEGIN_TKTEDIT<br />\n", -1); ticket_init(); getAllTicketFields(); initializeVariablesFromCGI(); initializeVariablesFromDb(); - @ <form method="POST" action="%s(g.zBaseURL)/%s(g.zPath)"> - @ <input type="hidden" name="name" value="%s(zName)"> + @ <form method="post" action="%s(g.zBaseURL)/%s(g.zPath)"><p> + @ <input type="hidden" name="name" value="%s(zName)" /> login_insert_csrf_secret(); + @ </p> zScript = ticket_editpage_code(); Th_Store("login", g.zLogin); Th_Store("date", db_text(0, "SELECT datetime('now')")); Th_CreateCommand(g.interp, "append_field", appendRemarkCmd, 0, 0); Th_CreateCommand(g.interp, "submit_ticket", submitTicketCmd, (void*)&zName,0); @@ -619,11 +633,11 @@ return 0; } /* ** WEBPAGE: tkttimeline -** URL: /tkttimeline?name=TICKETUUID&y=TYPE +** URL: /tkttimeline?name=TICKETUUID&y=TYPE ** ** Show the change history for a single ticket in timeline format. */ void tkttimeline_page(void){ Stmt q; @@ -639,11 +653,11 @@ if( !g.okHistory || !g.okRdTkt ){ login_needed(); return; } zUuid = PD("name",""); zType = PD("y","a"); if( zType[0]!='c' ){ style_submenu_element("Check-ins", "Check-ins", - "%s/tkttimeline?name=%T&y=ci", g.zTop, zUuid); + "%s/tkttimeline?name=%T&y=ci", g.zTop, zUuid); }else{ style_submenu_element("Timeline", "Timeline", "%s/tkttimeline?name=%T", g.zTop, zUuid); } style_submenu_element("History", "History", @@ -739,12 +753,11 @@ " AND blob.rid=attachid" " ORDER BY 1 DESC", tagid, tagid ); while( db_step(&q)==SQLITE_ROW ){ - Blob content; - Manifest m; + Manifest *pTicket; char zShort[12]; const char *zDate = db_column_text(&q, 0); int rid = db_column_int(&q, 1); const char *zChngUuid = db_column_text(&q, 2); const char *zFile = db_column_text(&q, 4); @@ -763,22 +776,22 @@ @ [<a href="%s(g.zTop)/artifact/%T(zChngUuid)">%s(zShort)</a>] @ (rid %d(rid)) by hyperlink_to_user(zUser,zDate," on"); hyperlink_to_date(zDate, ".</p>"); }else{ - content_get(rid, &content); - if( manifest_parse(&m, &content) && m.type==CFTYPE_TICKET ){ + pTicket = manifest_get(rid, CFTYPE_TICKET); + if( pTicket ){ @ @ <p>Ticket change @ [<a href="%s(g.zTop)/artifact/%T(zChngUuid)">%s(zShort)</a>] @ (rid %d(rid)) by - hyperlink_to_user(m.zUser,zDate," on"); + hyperlink_to_user(pTicket->zUser,zDate," on"); hyperlink_to_date(zDate, ":"); - ticket_output_change_artifact(&m); @ </p> + ticket_output_change_artifact(pTicket); } - manifest_clear(&m); + manifest_destroy(pTicket); } } db_finalize(&q); style_footer(); } @@ -820,5 +833,239 @@ } blob_reset(&val); } @ </ol> } + +/* +** COMMAND: ticket +** Usage: %fossil ticket SUBCOMMAND ... +** +** Run various subcommands to control tickets +** +** %fossil ticket show (REPORTTITLE|REPORTNR) ?TICKETFILTER? ?options? +** +** options can be: +** ?-l|--limit LIMITCHAR? +** ?-q|--quote? +** ?-R|--repository FILE? +** +** Run the ticket report, identified by the report format title +** used in the gui. The data is written as flat file on stdout, +** using "," as separator. The seperator "," can be changed using +** the -l or --limit option. +** If TICKETFILTER is given on the commandline, the query is +** limited with a new WHERE-condition. +** example: Report lists a column # with the uuid +** TICKETFILTER may be [#]='uuuuuuuuu' +** example: Report only lists rows with status not open +** TICKETFILTER: status != 'open' +** If the option -q|--quote is used, the tickets are encoded by +** quoting special chars(space -> \\s, tab -> \\t, newline -> \\n, +** cr -> \\r, formfeed -> \\f, vtab -> \\v, nul -> \\0, \\ -> \\\\). +** Otherwise, the simplified encoding as on the show report raw +** page in the gui is used. +** +** Instead of the report title its possible to use the report +** number. Using the special report number 0 list all columns, +** defined in the ticket table. +** +** %fossil ticket list fields +** +** list all fields, defined for ticket in the fossil repository +** +** %fossil ticket list reports +** +** list all ticket reports, defined in the fossil repository +** +** %fossil ticket set TICKETUUID FIELD VALUE ?FIELD VALUE .. ? ?-q|--quote? +** %fossil ticket change TICKETUUID FIELD VALUE ?FIELD VALUE .. ? ?-q|--quote? +** +** change ticket identified by TICKETUUID and set the value of +** field FIELD to VALUE. Valid field descriptions are: +** status, type, severity, priority, resolution, +** foundin, private_contact, resolution, title or comment +** Field names given above are the ones, defined in a standard +** fossil environment. If you have added, deleted columns, you +** change the all your configured columns. +** You can use more than one field/value pair on the commandline. +** Using -q|--quote enables the special character decoding as +** in "ticket show". So it's possible, to set multiline text or +** text with special characters. +** +** %fossil ticket add FIELD VALUE ?FIELD VALUE .. ? ?-q|--quote? +** +** like set, but create a new ticket with the given values. +** +** The values in set|add are not validated against the definitions +** given in "Ticket Common Script". +*/ +void ticket_cmd(void){ + int n; + + /* do some ints, we want to be inside a checkout */ + db_find_and_open_repository(1); + user_select(); + /* + ** Check that the user exists. + */ + if( !db_exists("SELECT 1 FROM user WHERE login=%Q", g.zLogin) ){ + fossil_fatal("no such user: %s", g.zLogin); + } + + if( g.argc<3 ){ + usage("add|fieldlist|set|show"); + }else{ + n = strlen(g.argv[2]); + if( n==1 && g.argv[2][0]=='s' ){ + /* set/show cannot be distinguished, so show the usage */ + usage("add|fieldlist|set|show"); + }else if( strncmp(g.argv[2],"list",n)==0 ){ + if( g.argc==3 ){ + usage("list fields|reports"); + }else{ + n = strlen(g.argv[3]); + if( !strncmp(g.argv[3],"fields",n) ){ + /* simply show all field names */ + int i; + + /* read all available ticket fields */ + getAllTicketFields(); + for(i=0; i<nField; i++){ + printf("%s\n",azField[i]); + } + }else if( !strncmp(g.argv[3],"reports",n) ){ + rpt_list_reports(); + }else{ + fossil_fatal("unknown ticket list option '%s'!",g.argv[3]); + } + } + }else{ + /* add a new ticket or set fields on existing tickets */ + tTktShowEncoding tktEncoding; + + tktEncoding = find_option("quote","q",0) ? tktFossilize : tktNoTab; + + if( strncmp(g.argv[2],"show",n)==0 ){ + if( g.argc==3 ){ + usage("show REPORTNR"); + }else{ + const char *zRep = 0; + const char *zSep = 0; + const char *zFilterUuid = 0; + + zSep = find_option("limit","l",1); + zRep = g.argv[3]; + if( !strcmp(zRep,"0") ){ + zRep = 0; + } + if( g.argc>4 ){ + zFilterUuid = g.argv[4]; + } + + rptshow( zRep, zSep, zFilterUuid, tktEncoding ); + + } + }else{ + /* add a new ticket or update an existing ticket */ + enum { set,add,err } eCmd = err; + int i = 0; + int rid; + const char *zTktUuid = 0; + Blob tktchng, cksum; + + /* get command type (set/add) and get uuid, if needed for set */ + if( strncmp(g.argv[2],"set",n)==0 || strncmp(g.argv[2],"change",n)==0 ){ + eCmd = set; + if( g.argc==3 ){ + usage("set TICKETUUID"); + } + zTktUuid = db_text(0, + "SELECT tkt_uuid FROM ticket WHERE tkt_uuid GLOB '%s*'", g.argv[3] + ); + if( !zTktUuid ){ + fossil_fatal("unknown ticket: '%s'!",g.argv[3]); + } + i=4; + }else if( strncmp(g.argv[2],"add",n)==0 ){ + eCmd = add; + i = 3; + zTktUuid = db_text(0, "SELECT lower(hex(randomblob(20)))"); + } + /* none of set/add, so show the usage! */ + if( eCmd==err ){ + usage("add|fieldlist|set|show"); + } + + /* read all given ticket field/value pairs from command line */ + if( i==g.argc ){ + fossil_fatal("empty %s command aborted!",g.argv[2]); + } + getAllTicketFields(); + /* read commandline and assign fields in the azValue array */ + while( i<g.argc ){ + char *zFName; + char *zFValue; + int j; + + zFName = g.argv[i++]; + if( i==g.argc ){ + fossil_fatal("missing value for '%s'!",zFName); + } + zFValue = g.argv[i++]; + j = fieldId(zFName); + if( tktEncoding == tktFossilize ){ + zFValue=mprintf("%s",zFValue); + defossilize(zFValue); + } + if( j == -1 ){ + fossil_fatal("unknown field name '%s'!",zFName); + }else{ + azValue[j] = zFValue; + } + } + + /* now add the needed artifacts to the repository */ + blob_zero(&tktchng); + { /* add the time to the ticket manifest */ + char *zDate; + + zDate = db_text(0, "SELECT datetime('now')"); + zDate[10] = 'T'; + blob_appendf(&tktchng, "D %s\n", zDate); + free(zDate); + } + /* append defined elements */ + for(i=0; i<nField; i++){ + char *zValue; + + zValue = azValue[i]; + if( azValue[i] && azValue[i][0] ){ + if( strncmp(azField[i], "private_", 8)==0 ){ + zValue = db_conceal(zValue, strlen(zValue)); + blob_appendf(&tktchng, "J %s %s\n", azField[i], zValue); + }else{ + blob_appendf(&tktchng, "J %s %#F\n", + azField[i], strlen(zValue), zValue); + } + if( tktEncoding == tktFossilize ){ + free(azValue[i]); + } + } + } + blob_appendf(&tktchng, "K %s\n", zTktUuid); + blob_appendf(&tktchng, "U %F\n", g.zLogin); + md5sum_blob(&tktchng, &cksum); + blob_appendf(&tktchng, "Z %b\n", &cksum); + rid = content_put(&tktchng, 0, 0); + if( rid==0 ){ + fossil_panic("trouble committing ticket: %s", g.zErrMsg); + } + manifest_crosslink_begin(); + manifest_crosslink(rid, &tktchng); + manifest_crosslink_end(); + printf("ticket %s succeeded for UID %s\n", + (eCmd==set?"set":"add"),zTktUuid); + } + } + } +} Index: src/tktsetup.c ================================================================== --- src/tktsetup.c +++ src/tktsetup.c @@ -123,28 +123,28 @@ z = zDfltValue; }else if( isSubmit ){ char *zErr = 0; login_verify_csrf_secret(); if( xText && (zErr = xText(z))!=0 ){ - @ <p><font color="red"><b>ERROR: %h(zErr)</b></font></p> + @ <p class="tktsetupError">ERROR: %h(zErr)</p> }else{ db_set(zDbField, z, 0); if( xRebuild ) xRebuild(); cgi_redirect("tktsetup"); } } - @ <form action="%s(g.zBaseURL)/%s(g.zPath)" method="POST"> + @ <form action="%s(g.zBaseURL)/%s(g.zPath)" method="post"><div> login_insert_csrf_secret(); @ <p>%s(zDesc)</p> @ <textarea name="x" rows="%d(height)" cols="80">%h(z)</textarea> - @ <blockquote> - @ <input type="submit" name="submit" value="Apply Changes"> - @ <input type="submit" name="clear" value="Revert To Default"> - @ <input type="submit" name="setup" value="Cancel"> - @ </blockquote> - @ </form> - @ <hr> + @ <blockquote><p> + @ <input type="submit" name="submit" value="Apply Changes" /> + @ <input type="submit" name="clear" value="Revert To Default" /> + @ <input type="submit" name="setup" value="Cancel" /> + @ </p></blockquote> + @ </div></form> + @ <hr /> @ <h2>Default %s(zTitle)</h2> @ <blockquote><pre> @ %h(zDfltValue) @ </pre></blockquote> style_footer(); @@ -153,13 +153,13 @@ /* ** WEBPAGE: tktsetup_tab */ void tktsetup_tab_page(void){ static const char zDesc[] = - @ <p>Enter a valid CREATE TABLE statement for the "ticket" table. The + @ Enter a valid CREATE TABLE statement for the "ticket" table. The @ table must contain columns named "tkt_id", "tkt_uuid", and "tkt_mtime" - @ with an unique index on "tkt_uuid" and "tkt_mtime".</p> + @ with an unique index on "tkt_uuid" and "tkt_mtime". ; tktsetup_generic( "Ticket Table Schema", "ticket-table", zDefaultTicketTable, @@ -229,12 +229,12 @@ /* ** WEBPAGE: tktsetup_com */ void tktsetup_com_page(void){ static const char zDesc[] = - @ <p>Enter TH1 script that initializes variables prior to generating - @ any of the ticket view, edit, or creation pages.</p> + @ Enter TH1 script that initializes variables prior to generating + @ any of the ticket view, edit, or creation pages. ; tktsetup_generic( "Ticket Common Script", "ticket-common", zDefaultTicketCommon, @@ -250,80 +250,80 @@ @ if {[info exists submit]} { @ set status Open @ submit_ticket @ } @ </th1> -@ <h1 align="center">Enter A New Ticket</h1> +@ <h1 style="text-align: center;">Enter A New Ticket</h1> @ <table cellpadding="5"> @ <tr> @ <td colspan="2"> -@ Enter a one-line summary of the ticket:<br> -@ <input type="text" name="title" size="60" value="$<title>"> +@ Enter a one-line summary of the ticket:<br /> +@ <input type="text" name="title" size="60" value="$<title>" /> @ </td> @ </tr> @ @ <tr> -@ <td align="right">Type: +@ <td style="text-align: center;">Type: @ <th1>combobox type $type_choices 1</th1> @ </td> @ <td>What type of ticket is this?</td> @ </tr> @ @ <tr> -@ <td align="right">Version: -@ <input type="text" name="foundin" size="20" value="$<foundin>"> +@ <td style="text-align: center;">Version: +@ <input type="text" name="foundin" size="20" value="$<foundin>" /> @ </td> @ <td>In what version or build number do you observe the problem?</td> @ </tr> @ @ <tr> -@ <td align="right">Severity: +@ <td style="text-align: center;">Severity: @ <th1>combobox severity $severity_choices 1</th1> @ </td> @ <td>How debilitating is the problem? How badly does the problem @ affect the operation of the product?</td> @ </tr> @ @ <tr> -@ <td align="right">EMail: -@ <input type="text" name="private_contact" value="$<private_contact>" size="30"> +@ <td style="text-align: center;">EMail: +@ <input type="text" name="private_contact" value="$<private_contact>" size="30" /> @ </td> -@ <td><u>Not publicly visible</u>. Used by developers to contact you with -@ questions.</td> +@ <td><span style="text-decoration: underline;">Not publicly visible</span>. +@ Used by developers to contact you with questions.</td> @ </tr> @ @ <tr> @ <td colspan="2"> @ Enter a detailed description of the problem. @ For code defects, be sure to provide details on exactly how @ the problem can be reproduced. Provide as much detail as @ possible. -@ <br> +@ <br /> @ <th1>set nline [linecount $comment 50 10]</th1> @ <textarea name="comment" cols="80" rows="$nline" -@ wrap="virtual" class="wikiedit">$<comment></textarea><br> -@ <input type="submit" name="preview" value="Preview"> +@ wrap="virtual" class="wikiedit">$<comment></textarea><br /> +@ <input type="submit" name="preview" value="Preview" /></td> @ </tr> @ @ <th1>enable_output [info exists preview]</th1> @ <tr><td colspan="2"> -@ Description Preview:<br><hr> +@ Description Preview:<br /><hr /> @ <th1>wiki $comment</th1> -@ <hr> +@ <hr /> @ </td></tr> @ <th1>enable_output 1</th1> @ @ <tr> -@ <td align="right"> -@ <input type="submit" name="submit" value="Submit"> +@ <td style="text-align: center;"> +@ <input type="submit" name="submit" value="Submit" /> @ </td> @ <td>After filling in the information above, press this button to create @ the new ticket</td> @ </tr> @ <tr> -@ <td align="right"> -@ <input type="submit" name="cancel" value="Cancel"> +@ <td style="text-align: center;"> +@ <input type="submit" name="cancel" value="Cancel" /> @ </td> @ <td>Abandon and forget this ticket</td> @ </tr> @ </table> ; @@ -338,12 +338,12 @@ /* ** WEBPAGE: tktsetup_newpage */ void tktsetup_newpage_page(void){ static const char zDesc[] = - @ <p>Enter HTML with embedded TH1 script that will render the "new ticket" - @ page</p> + @ Enter HTML with embedded TH1 script that will render the "new ticket" + @ page ; tktsetup_generic( "HTML For New Tickets", "ticket-newpage", zDefaultNew, @@ -354,51 +354,50 @@ ); } static const char zDefaultView[] = @ <table cellpadding="5"> -@ <tr><td align="right">Ticket UUID:</td><td bgcolor="#d0d0d0" colspan="3"> -@ $<tkt_uuid> -@ </td></tr> -@ <tr><td align="right">Title:</td> -@ <td bgcolor="#d0d0d0" colspan="3" valign="top"> +@ <tr><td class="tktDspLabel">Ticket UUID:</td> +@ <td class="tktDspValue" colspan="3">$<tkt_uuid></td></tr> +@ <tr><td class="tktDspLabel">Title:</td> +@ <td class="tktDspValue" colspan="3"> @ <th1>wiki $title</th1> @ </td></tr> -@ <tr><td align="right">Status:</td><td bgcolor="#d0d0d0"> +@ <tr><td class="tktDspLabel">Status:</td><td class="tktDspValue"> @ $<status> @ </td> -@ <td align="right">Type:</td><td bgcolor="#d0d0d0"> +@ <td class="tktDspLabel">Type:</td><td class="tktDspValue"> @ $<type> @ </td></tr> -@ <tr><td align="right">Severity:</td><td bgcolor="#d0d0d0"> +@ <tr><td class="tktDspLabel">Severity:</td><td class="tktDspValue"> @ $<severity> @ </td> -@ <td align="right">Priority:</td><td bgcolor="#d0d0d0"> +@ <td class="tktDspLabel">Priority:</td><td class="tktDspValue"> @ $<priority> @ </td></tr> -@ <tr><td align="right">Subsystem:</td><td bgcolor="#d0d0d0"> +@ <tr><td class="tktDspLabel">Subsystem:</td><td class="tktDspValue"> @ $<subsystem> @ </td> -@ <td align="right">Resolution:</td><td bgcolor="#d0d0d0"> +@ <td class="tktDspLabel">Resolution:</td><td class="tktDspValue"> @ $<resolution> @ </td></tr> -@ <tr><td align="right">Last Modified:</td><td bgcolor="#d0d0d0"> +@ <tr><td class="tktDspLabel">Last Modified:</td><td class="tktDspValue"> @ $<tkt_datetime> @ </td> @ <th1>enable_output [hascap e]</th1> -@ <td align="right">Contact:</td><td bgcolor="#d0d0d0"> +@ <td class="tktDspLabel">Contact:</td><td class="tktDspValue"> @ $<private_contact> @ </td> @ <th1>enable_output 1</th1> @ </tr> -@ <tr><td align="right">Version Found In:</td> -@ <td colspan="3" valign="top" bgcolor="#d0d0d0"> +@ <tr><td class="tktDspLabel">Version Found In:</td> +@ <td colspan="3" valign="top" class="tktDspValue"> @ $<foundin> @ </td></tr> @ <tr><td>Description & Comments:</td></tr> -@ <tr><td colspan="4" bgcolor="#d0d0d0"> -@ <span bgcolor="#d0d0d0"><th1>wiki $comment</th1></span> +@ <tr><td colspan="4" class="tktDspValue"> +@ <th1>wiki $comment</th1> @ </td></tr> @ </table> ; @@ -412,12 +411,11 @@ /* ** WEBPAGE: tktsetup_viewpage */ void tktsetup_viewpage_page(void){ static const char zDesc[] = - @ <p>Enter HTML with embedded TH1 script that will render the "view ticket" - @ page</p> + @ Enter HTML with embedded TH1 script that will render the "view ticket" page ; tktsetup_generic( "HTML For Viewing Tickets", "ticket-viewpage", zDefaultView, @@ -432,51 +430,51 @@ @ <th1> @ if {![info exists username]} {set username $login} @ if {[info exists submit]} { @ if {[info exists cmappnd]} { @ if {[string length $cmappnd]>0} { -@ set ctxt "\n\n<hr><i>[htmlize $login]" +@ set ctxt "\n\n<hr /><i>[htmlize $login]" @ if {$username ne $login} { @ set ctxt "$ctxt claiming to be [htmlize $username]" @ } -@ set ctxt "$ctxt added on [date]:</i><br>\n$cmappnd" +@ set ctxt "$ctxt added on [date]:</i><br />\n$cmappnd" @ append_field comment $ctxt @ } @ } @ submit_ticket @ } @ </th1> @ <table cellpadding="5"> -@ <tr><td align="right">Title:</td><td> -@ <input type="text" name="title" value="$<title>" size="60"> +@ <tr><td class="tktDspLabel">Title:</td><td> +@ <input type="text" name="title" value="$<title>" size="60" /> @ </td></tr> -@ <tr><td align="right">Status:</td><td> +@ <tr><td class="tktDspLabel">Status:</td><td> @ <th1>combobox status $status_choices 1</th1> @ </td></tr> -@ <tr><td align="right">Type:</td><td> +@ <tr><td class="tktDspLabel">Type:</td><td> @ <th1>combobox type $type_choices 1</th1> @ </td></tr> -@ <tr><td align="right">Severity:</td><td> +@ <tr><td class="tktDspLabel">Severity:</td><td> @ <th1>combobox severity $severity_choices 1</th1> @ </td></tr> -@ <tr><td align="right">Priority:</td><td> +@ <tr><td class="tktDspLabel">Priority:</td><td> @ <th1>combobox priority $priority_choices 1</th1> @ </td></tr> -@ <tr><td align="right">Resolution:</td><td> +@ <tr><td class="tktDspLabel">Resolution:</td><td> @ <th1>combobox resolution $resolution_choices 1</th1> @ </td></tr> -@ <tr><td align="right">Subsystem:</td><td> +@ <tr><td class="tktDspLabel">Subsystem:</td><td> @ <th1>combobox subsystem $subsystem_choices 1</th1> @ </td></tr> @ <th1>enable_output [hascap e]</th1> -@ <tr><td align="right">Contact:</td><td> +@ <tr><td class="tktDspLabel">Contact:</td><td> @ <input type="text" name="private_contact" size="40" -@ value="$<private_contact>"> +@ value="$<private_contact>" /> @ </td></tr> @ <th1>enable_output 1</th1> -@ <tr><td align="right">Version Found In:</td><td> -@ <input type="text" name="foundin" size="50" value="$<foundin>"> +@ <tr><td class="tktDspLabel">Version Found In:</td><td> +@ <input type="text" name="foundin" size="50" value="$<foundin>" /> @ </td></tr> @ <tr><td colspan="2"> @ <th1> @ if {![info exists eall]} {set eall 0} @ if {[info exists aonlybtn]} {set eall 0} @@ -484,45 +482,45 @@ @ if {![hascap w]} {set eall 0} @ if {![info exists cmappnd]} {set cmappnd {}} @ set nline [linecount $comment 15 10] @ enable_output $eall @ </th1> -@ Description And Comments:<br> +@ Description And Comments:<br /> @ <textarea name="comment" cols="80" rows="$nline" -@ wrap="virtual" class="wikiedit">$<comment></textarea><br> -@ <input type="hidden" name="eall" value="1"> -@ <input type="submit" name="aonlybtn" value="Append Remark"> -@ <input type="submit" name="preview1btn" value="Preview"> +@ wrap="virtual" class="wikiedit">$<comment></textarea><br /> +@ <input type="hidden" name="eall" value="1" /> +@ <input type="submit" name="aonlybtn" value="Append Remark" /> +@ <input type="submit" name="preview1btn" value="Preview" /> @ <th1>enable_output [expr {!$eall}]</th1> @ Append Remark from -@ <input type="text" name="username" value="$<username>" size="30">:<br> +@ <input type="text" name="username" value="$<username>" size="30" />:<br /> @ <textarea name="cmappnd" cols="80" rows="15" -@ wrap="virtual" class="wikiedit">$<cmappnd></textarea><br> +@ wrap="virtual" class="wikiedit">$<cmappnd></textarea><br /> @ <th1>enable_output [expr {[hascap w] && !$eall}]</th1> -@ <input type="submit" name="eallbtn" value="Edit All"> +@ <input type="submit" name="eallbtn" value="Edit All" /> @ <th1>enable_output [expr {!$eall}]</th1> -@ <input type="submit" name="preview2btn" value="Preview"> +@ <input type="submit" name="preview2btn" value="Preview" /> @ <th1>enable_output 1</th1> @ </td></tr> @ @ <th1>enable_output [info exists preview1btn]</th1> @ <tr><td colspan="2"> -@ Description Preview:<br><hr> +@ Description Preview:<br /><hr /> @ <th1>wiki $comment</th1> -@ <hr> +@ <hr /> @ </td></tr> @ <th1>enable_output [info exists preview2btn]</th1> @ <tr><td colspan="2"> -@ Description Preview:<br><hr> +@ Description Preview:<br /><hr /> @ <th1>wiki $cmappnd</th1> -@ <hr> +@ <hr /> @ </td></tr> @ <th1>enable_output 1</th1> @ @ <tr><td align="right"></td><td> -@ <input type="submit" name="submit" value="Submit Changes"> -@ <input type="submit" name="cancel" value="Cancel"> +@ <input type="submit" name="submit" value="Submit Changes" /> +@ <input type="submit" name="cancel" value="Cancel" /> @ </td></tr> @ </table> ; /* @@ -535,12 +533,11 @@ /* ** WEBPAGE: tktsetup_editpage */ void tktsetup_editpage_page(void){ static const char zDesc[] = - @ <p>Enter HTML with embedded TH1 script that will render the "edit ticket" - @ page</p> + @ Enter HTML with embedded TH1 script that will render the "edit ticket" page ; tktsetup_generic( "HTML For Editing Tickets", "ticket-editpage", zDefaultEdit, @@ -585,12 +582,11 @@ /* ** WEBPAGE: tktsetup_reportlist */ void tktsetup_reportlist(void){ static const char zDesc[] = - @ <p>Enter HTML with embedded TH1 script that will render the "report list" - @ page</p> + @ Enter HTML with embedded TH1 script that will render the "report list" page ; tktsetup_generic( "HTML For Report List", "ticket-reportlist", zDefaultReportList, @@ -633,13 +629,13 @@ /* ** WEBPAGE: tktsetup_rpttplt */ void tktsetup_rpttplt_page(void){ static const char zDesc[] = - @ <p>Enter the default ticket report format template. This is the + @ Enter the default ticket report format template. This is the @ the template report format that initially appears when creating a - @ new ticket summary report.</p> + @ new ticket summary report. ; tktsetup_generic( "Default Report Template", "ticket-report-template", zDefaultReport, @@ -674,13 +670,13 @@ /* ** WEBPAGE: tktsetup_keytplt */ void tktsetup_keytplt_page(void){ static const char zDesc[] = - @ <p>Enter the default ticket report color-key template. This is the + @ Enter the default ticket report color-key template. This is the @ the color-key that initially appears when creating a - @ new ticket summary report.</p> + @ new ticket summary report. ; tktsetup_generic( "Default Report Color-Key Template", "ticket-key-template", zDefaultKey, @@ -703,34 +699,34 @@ if( P("setup") ){ cgi_redirect("tktsetup"); } style_header("Ticket Display On Timelines"); db_begin_transaction(); - @ <form action="%s(g.zBaseURL)/tktsetup_timeline" method="POST"> + @ <form action="%s(g.zBaseURL)/tktsetup_timeline" method="post"><div> login_insert_csrf_secret(); - @ <hr> + @ <hr /> entry_attribute("Ticket Title", 40, "ticket-title-expr", "t", "title"); @ <p>An SQL expression in a query against the TICKET table that will @ return the title of the ticket for display purposes.</p> - @ <hr> + @ <hr /> entry_attribute("Ticket Status", 40, "ticket-status-column", "s", "status"); @ <p>The name of the column in the TICKET table that contains the ticket @ status in human-readable form. Case sensitive.</p> - @ <hr> + @ <hr /> entry_attribute("Ticket Closed", 40, "ticket-closed-expr", "c", "status='Closed'"); @ <p>An SQL expression that evaluates to true in a TICKET table query if @ the ticket is closed.</p> - @ <hr> + @ <hr /> @ <p> - @ <input type="submit" name="submit" value="Apply Changes"> - @ <input type="submit" name="setup" value="Cancel"> + @ <input type="submit" name="submit" value="Apply Changes" /> + @ <input type="submit" name="setup" value="Cancel" /> @ </p> - @ </form> + @ </div></form> db_end_transaction(0); style_footer(); } Index: src/translate.c ================================================================== --- src/translate.c +++ src/translate.c @@ -66,15 +66,16 @@ /* ** Translate the input stream into the output stream */ static void trans(FILE *in, FILE *out){ - int i, j, k; /* Loop counters */ - char c1, c2; /* Characters used to start a comment */ - int lastWasEq = 0; /* True if last non-whitespace character was "=" */ - char zLine[2000]; /* A single line of input */ - char zOut[4000]; /* The input line translated into appropriate output */ + int i, j, k; /* Loop counters */ + char c1, c2; /* Characters used to start a comment */ + int lastWasEq = 0; /* True if last non-whitespace character was "=" */ + int lastWasComma = 0; /* True if last non-whitespace character was "," */ + char zLine[2000]; /* A single line of input */ + char zOut[4000]; /* The input line translated into appropriate output */ c1 = c2 = '-'; while( fgets(zLine, sizeof(zLine), in) ){ for(i=0; zLine[i] && isspace(zLine[i]); i++){} if( zLine[i]!='@' ){ @@ -85,14 +86,16 @@ c1 = zLine[14]; c2 = zLine[15]; } i += strlen(&zLine[i]); while( i>0 && isspace(zLine[i-1]) ){ i--; } - lastWasEq = i>0 && zLine[i-1]=='='; - }else if( lastWasEq ){ + lastWasEq = i>0 && zLine[i-1]=='='; + lastWasComma = i>0 && zLine[i-1]==','; + }else if( lastWasEq || lastWasComma){ /* If the last non-whitespace character before the first @ was - ** an "=" then generate a string literal. But skip comments + ** an "="(var init/set) or a ","(const definition in list) then + ** generate a string literal. But skip comments ** consisting of all text between c1 and c2 (default "--") ** and end of line. */ int indent, omitline; i++; Index: src/update.c ================================================================== --- src/update.c +++ src/update.c @@ -24,11 +24,11 @@ /* ** Return true if artifact rid is a version */ int is_a_version(int rid){ - return db_exists("SELECT 1 FROM plink WHERE cid=%d", rid); + return db_exists("SELECT 1 FROM event WHERE objid=%d AND type='ci'", rid); } /* ** COMMAND: update ** @@ -111,10 +111,11 @@ tid = db_int(0, "SELECT rid FROM leaves, event" " WHERE event.objid=leaves.rid" " ORDER BY event.mtime DESC"); } + if( tid==vid ) return; /* Nothing to update */ db_begin_transaction(); vfile_check_signature(vid, 1); if( !nochangeFlag ) undo_begin(); load_vfile_from_rid(tid); @@ -278,10 +279,11 @@ db_end_transaction(1); /* With --nochange, rollback changes */ }else{ if( g.argc<=3 ){ /* All files updated. Shift the current checkout to the target. */ db_multi_exec("DELETE FROM vfile WHERE vid!=%d", tid); + checkout_set_all_exe(vid); manifest_to_disk(tid); db_lset_int("checkout", tid); }else{ /* A subset of files have been checked out. Keep the current ** checkout unchanged. */ @@ -301,13 +303,13 @@ const char *revision, /* The checkin containing the file */ const char *file, /* Full treename of the file */ Blob *content, /* Put the content here */ int errCode /* Error code if file not found. Panic if 0. */ ){ - Blob mfile; - Manifest m; - int i, rid=0; + Manifest *pManifest; + ManifestFile *pFile; + int rid=0; if( revision ){ rid = name_to_rid(revision); }else{ rid = db_lget_int("checkout", 0); @@ -314,21 +316,22 @@ } if( !is_a_version(rid) ){ if( errCode>0 ) return errCode; fossil_fatal("no such checkin: %s", revision); } - content_get(rid, &mfile); + pManifest = manifest_get(rid, CFTYPE_MANIFEST); - if( manifest_parse(&m, &mfile) ){ - for(i=0; i<m.nFile; i++){ - if( strcmp(m.aFile[i].zName, file)==0 ){ - rid = uuid_to_rid(m.aFile[i].zUuid, 0); - manifest_clear(&m); + if( pManifest ){ + manifest_file_rewind(pManifest); + while( (pFile = manifest_file_next(pManifest,0))!=0 ){ + if( strcmp(pFile->zName, file)==0 ){ + rid = uuid_to_rid(pFile->zUuid, 0); + manifest_destroy(pManifest); return content_get(rid, content); } } - manifest_clear(&m); + manifest_destroy(pManifest); if( errCode<=0 ){ fossil_fatal("file %s does not exist in checkin: %s", file, revision); } }else if( errCode<=0 ){ fossil_panic("could not parse manifest for checkin: %s", revision); @@ -385,14 +388,15 @@ }else{ int vid; vid = db_lget_int("checkout", 0); vfile_check_signature(vid, 0); db_multi_exec( + "DELETE FROM vmerge;" "INSERT INTO torevert " "SELECT pathname" " FROM vfile " - " WHERE chnged OR deleted OR rid=0 OR pathname!=origname" + " WHERE chnged OR deleted OR rid=0 OR pathname!=origname;" ); } blob_zero(&record); db_prepare(&q, "SELECT name FROM torevert"); while( db_step(&q)==SQLITE_ROW ){ Index: src/url.c ================================================================== --- src/url.c +++ src/url.c @@ -17,18 +17,29 @@ ** ** This file contains code for parsing URLs that appear on the command-line */ #include "config.h" #include "url.h" + +/* +** Convert a string to lower-case. +*/ +static void url_tolower(char *z){ + while( *z ){ + *z = fossil_tolower(*z); + z++; + } +} /* ** Parse the given URL. Populate variables in the global "g" structure. ** ** g.urlIsFile True if FILE: ** g.urlIsHttps True if HTTPS: +** g.urlIsSsh True if SSH: ** g.urlProtocol "http" or "https" or "file" -** g.urlName Hostname for HTTP: or HTTPS:. Filename for FILE: +** g.urlName Hostname for HTTP:, HTTPS:, SSH:. Filename for FILE: ** g.urlPort TCP port number for HTTP or HTTPS. ** g.urlDfltPort Default TCP port number (80 or 443). ** g.urlPath Path name for HTTP or HTTPS. ** g.urlUser Userid. ** g.urlPasswd Password. @@ -35,76 +46,125 @@ ** g.urlHostname HOST:PORT or just HOST if port is the default. ** g.urlCanonical The URL in canonical form, omitting the password ** ** HTTP url format is: ** -** http://userid:password@host:port/path?query#fragment +** http://userid:password@host:port/path +** +** SSH url format is: +** +** ssh://userid:password@host:port/path?fossil=path/to/fossil.exe ** */ void url_parse(const char *zUrl){ int i, j, c; char *zFile = 0; - if( strncmp(zUrl, "http://", 7)==0 || strncmp(zUrl, "https://", 8)==0 ){ + if( strncmp(zUrl, "http://", 7)==0 + || strncmp(zUrl, "https://", 8)==0 + || strncmp(zUrl, "ssh://", 6)==0 + ){ int iStart; char *zLogin; + char *zExe; + g.urlIsFile = 0; if( zUrl[4]=='s' ){ g.urlIsHttps = 1; g.urlProtocol = "https"; g.urlDfltPort = 443; iStart = 8; + }else if( zUrl[0]=='s' ){ + g.urlIsSsh = 1; + g.urlProtocol = "ssh"; + g.urlDfltPort = 22; + g.urlFossil = "fossil"; + iStart = 6; }else{ g.urlIsHttps = 0; g.urlProtocol = "http"; g.urlDfltPort = 80; iStart = 7; } for(i=iStart; (c=zUrl[i])!=0 && c!='/' && c!='@'; i++){} if( c=='@' ){ + /* Parse up the user-id and password */ for(j=iStart; j<i && zUrl[j]!=':'; j++){} g.urlUser = mprintf("%.*s", j-iStart, &zUrl[iStart]); dehttpize(g.urlUser); if( j<i ){ g.urlPasswd = mprintf("%.*s", i-j-1, &zUrl[j+1]); dehttpize(g.urlPasswd); } + if( g.urlIsSsh && g.urlPasswd ){ + zLogin = mprintf("%t:*@", g.urlUser); + }else{ + zLogin = mprintf("%t@", g.urlUser); + } for(j=i+1; (c=zUrl[j])!=0 && c!='/' && c!=':'; j++){} g.urlName = mprintf("%.*s", j-i-1, &zUrl[i+1]); i = j; - zLogin = mprintf("%t@", g.urlUser); }else{ for(i=iStart; (c=zUrl[i])!=0 && c!='/' && c!=':'; i++){} g.urlName = mprintf("%.*s", i-iStart, &zUrl[iStart]); zLogin = mprintf(""); } - for(j=0; g.urlName[j]; j++){ g.urlName[j] = tolower(g.urlName[j]); } + url_tolower(g.urlName); if( c==':' ){ g.urlPort = 0; i++; - while( (c = zUrl[i])!=0 && isdigit(c) ){ + while( (c = zUrl[i])!=0 && fossil_isdigit(c) ){ g.urlPort = g.urlPort*10 + c - '0'; i++; } g.urlHostname = mprintf("%s:%d", g.urlName, g.urlPort); }else{ g.urlPort = g.urlDfltPort; g.urlHostname = g.urlName; } - g.urlPath = mprintf(&zUrl[i]); dehttpize(g.urlName); + g.urlPath = mprintf("%s", &zUrl[i]); + for(i=0; g.urlPath[i] && g.urlPath[i]!='?'; i++){} + if( g.urlPath[i] ){ + g.urlPath[i] = 0; + i++; + } + zExe = mprintf(""); + while( g.urlPath[i]!=0 ){ + char *zName, *zValue; + zName = &g.urlPath[i]; + zValue = zName; + while( g.urlPath[i] && g.urlPath[i]!='=' ){ i++; } + if( g.urlPath[i]=='=' ){ + g.urlPath[i] = 0; + i++; + zValue = &g.urlPath[i]; + while( g.urlPath[i] && g.urlPath[i]!='&' ){ i++; } + } + if( g.urlPath[i] ){ + g.urlPath[i] = 0; + i++; + } + if( strcmp(zName,"fossil")==0 ){ + g.urlFossil = zValue; + dehttpize(g.urlFossil); + zExe = mprintf("?fossil=%T", g.urlFossil); + } + } + dehttpize(g.urlPath); if( g.urlDfltPort==g.urlPort ){ g.urlCanonical = mprintf( - "%s://%s%T%T", - g.urlProtocol, zLogin, g.urlName, g.urlPath + "%s://%s%T%T%s", + g.urlProtocol, zLogin, g.urlName, g.urlPath, zExe ); }else{ g.urlCanonical = mprintf( - "%s://%s%T:%d%T", - g.urlProtocol, zLogin, g.urlName, g.urlPort, g.urlPath + "%s://%s%T:%d%T%s", + g.urlProtocol, zLogin, g.urlName, g.urlPort, g.urlPath, zExe ); } + if( g.urlIsSsh && g.urlPath[1] ) g.urlPath++; free(zLogin); }else if( strncmp(zUrl, "file:", 5)==0 ){ g.urlIsFile = 1; if( zUrl[5]=='/' && zUrl[6]=='/' ){ i = 7; @@ -150,19 +210,22 @@ } url_parse(g.argv[2]); for(i=0; i<2; i++){ printf("g.urlIsFile = %d\n", g.urlIsFile); printf("g.urlIsHttps = %d\n", g.urlIsHttps); + printf("g.urlIsSsh = %d\n", g.urlIsSsh); printf("g.urlProtocol = %s\n", g.urlProtocol); printf("g.urlName = %s\n", g.urlName); printf("g.urlPort = %d\n", g.urlPort); printf("g.urlDfltPort = %d\n", g.urlDfltPort); printf("g.urlHostname = %s\n", g.urlHostname); printf("g.urlPath = %s\n", g.urlPath); printf("g.urlUser = %s\n", g.urlUser); printf("g.urlPasswd = %s\n", g.urlPasswd); printf("g.urlCanonical = %s\n", g.urlCanonical); + printf("g.urlFossil = %s\n", g.urlFossil); + if( g.urlIsFile || g.urlIsSsh ) break; if( i==0 ){ printf("********\n"); url_enable_proxy("Using proxy: "); } } @@ -288,11 +351,11 @@ zName2 = 0; z = zValue2; if( z==0 ) continue; } blob_appendf(&p->url, "%s%s=%T", zSep, p->azName[i], z); - zSep = "&"; + zSep = "&"; } if( zName1 && zValue1 ){ blob_appendf(&p->url, "%s%s=%T", zSep, zName1, zValue1); } if( zName2 && zValue2 ){ Index: src/user.c ================================================================== --- src/user.c +++ src/user.c @@ -27,23 +27,26 @@ ** onto the end of a blob. */ static void strip_string(Blob *pBlob, char *z){ int i; blob_reset(pBlob); - while( isspace(*z) ){ z++; } + while( fossil_isspace(*z) ){ z++; } for(i=0; z[i]; i++){ if( z[i]=='\r' || z[i]=='\n' ){ - while( i>0 && isspace(z[i-1]) ){ i--; } + while( i>0 && fossil_isspace(z[i-1]) ){ i--; } z[i] = 0; break; } if( z[i]<' ' ) z[i] = ' '; } blob_append(pBlob, z, -1); } +#if defined(_WIN32) #ifdef __MINGW32__ +#include <conio.h> +#endif /* ** getpass for Windows */ static char *getpass(const char *prompt){ static char pwd[64]; Index: src/vfile.c ================================================================== --- src/vfile.c +++ src/vfile.c @@ -19,11 +19,15 @@ */ #include "config.h" #include "vfile.h" #include <assert.h> #include <sys/types.h> +#if defined(__DMC__) +#include "dirent.h" +#else #include <dirent.h> +#endif /* ** Given a UUID, return the corresponding record ID. If the UUID ** does not exist, then return 0. ** @@ -81,57 +85,38 @@ } } } /* -** Build a catalog of all files in a baseline. -** We scan the baseline file for lines of the form: -** -** F NAME UUID -** -** Each such line makes an entry in the VFILE table. +** Build a catalog of all files in a checkin. */ -void vfile_build(int vid, Blob *p){ +void vfile_build(int vid){ int rid; - char *zName, *zUuid; Stmt ins; - Blob line, token, name, uuid; - int seenHeader = 0; + Manifest *p; + ManifestFile *pFile; + db_begin_transaction(); vfile_verify_not_phantom(vid, 0, 0); + p = manifest_get(vid, CFTYPE_MANIFEST); + if( p==0 ) return; db_multi_exec("DELETE FROM vfile WHERE vid=%d", vid); db_prepare(&ins, "INSERT INTO vfile(vid,rid,mrid,pathname) " " VALUES(:vid,:id,:id,:name)"); db_bind_int(&ins, ":vid", vid); - while( blob_line(p, &line) ){ - char *z = blob_buffer(&line); - if( z[0]=='-' ){ - if( seenHeader ) break; - while( blob_line(p, &line)>2 ){} - if( blob_line(p, &line)==0 ) break; - } - seenHeader = 1; - if( z[0]!='F' || z[1]!=' ' ) continue; - blob_token(&line, &token); /* Skip the "F" token */ - if( blob_token(&line, &name)==0 ) break; - if( blob_token(&line, &uuid)==0 ) break; - zName = blob_str(&name); - defossilize(zName); - zUuid = blob_str(&uuid); - rid = uuid_to_rid(zUuid, 0); - vfile_verify_not_phantom(rid, zName, zUuid); - if( rid>0 && file_is_simple_pathname(zName) ){ - db_bind_int(&ins, ":id", rid); - db_bind_text(&ins, ":name", zName); - db_step(&ins); - db_reset(&ins); - } - blob_reset(&name); - blob_reset(&uuid); + manifest_file_rewind(p); + while( (pFile = manifest_file_next(p,0))!=0 ){ + rid = uuid_to_rid(pFile->zUuid, 0); + vfile_verify_not_phantom(rid, pFile->zName, pFile->zUuid); + db_bind_int(&ins, ":id", rid); + db_bind_text(&ins, ":name", pFile->zName); + db_step(&ins); + db_reset(&ins); } db_finalize(&ins); + manifest_destroy(p); db_end_transaction(0); } /* ** Check the file signature of the disk image for every VFILE of vid. @@ -365,10 +350,11 @@ } fseek(in, 0L, SEEK_END); sprintf(zBuf, " %ld\n", ftell(in)); fseek(in, 0L, SEEK_SET); md5sum_step_text(zBuf, -1); + /*printf("%s %s %s",md5sum_current_state(),zName,zBuf); fflush(stdout);*/ for(;;){ int n; n = fread(zBuf, 1, sizeof(zBuf), in); if( n<=0 ) break; md5sum_step_text(zBuf, n); @@ -418,10 +404,11 @@ int rid = db_column_int(&q, 1); md5sum_step_text(zName, -1); content_get(rid, &file); sprintf(zBuf, " %d\n", blob_size(&file)); md5sum_step_text(zBuf, -1); + /*printf("%s %s %s",md5sum_current_state(),zName,zBuf); fflush(stdout);*/ md5sum_step_blob(&file); blob_reset(&file); } db_finalize(&q); md5sum_finish(pOut); @@ -434,37 +421,43 @@ ** ** If pManOut is not NULL then fill it with the checksum found in the ** "R" card near the end of the manifest. */ void vfile_aggregate_checksum_manifest(int vid, Blob *pOut, Blob *pManOut){ - int i, fid; - Blob file, mfile; - Manifest m; + int fid; + Blob file; + Manifest *pManifest; + ManifestFile *pFile; char zBuf[100]; blob_zero(pOut); if( pManOut ){ blob_zero(pManOut); } db_must_be_within_tree(); - content_get(vid, &mfile); - if( manifest_parse(&m, &mfile)==0 ){ + pManifest = manifest_get(vid, CFTYPE_MANIFEST); + if( pManifest==0 ){ fossil_panic("manifest file (%d) is malformed", vid); } - for(i=0; i<m.nFile; i++){ - fid = uuid_to_rid(m.aFile[i].zUuid, 0); - md5sum_step_text(m.aFile[i].zName, -1); + manifest_file_rewind(pManifest); + while( (pFile = manifest_file_next(pManifest,0))!=0 ){ + fid = uuid_to_rid(pFile->zUuid, 0); + md5sum_step_text(pFile->zName, -1); content_get(fid, &file); sprintf(zBuf, " %d\n", blob_size(&file)); md5sum_step_text(zBuf, -1); md5sum_step_blob(&file); blob_reset(&file); } if( pManOut ){ - blob_append(pManOut, m.zRepoCksum, -1); + if( pManifest->zRepoCksum ){ + blob_append(pManOut, pManifest->zRepoCksum, -1); + }else{ + blob_zero(pManOut); + } } - manifest_clear(&m); + manifest_destroy(pManifest); md5sum_finish(pOut); } /* ** COMMAND: test-agg-cksum Index: src/wiki.c ================================================================== --- src/wiki.c +++ src/wiki.c @@ -48,15 +48,15 @@ /* ** Output rules for well-formed wiki pages */ static void well_formed_wiki_name_rules(void){ @ <ul> - @ <li> Must not begin or end with a space. + @ <li> Must not begin or end with a space.</li> @ <li> Must not contain any control characters, including tab or - @ newline. - @ <li> Must not have two or more spaces in a row internally. - @ <li> Must be between 3 and 100 characters in length. + @ newline.</li> + @ <li> Must not have two or more spaces in a row internally.</li> + @ <li> Must be between 3 and 100 characters in length.</li> @ </ul> } /* ** Check a wiki name. If it is not well-formed, then issue an error @@ -63,12 +63,12 @@ ** and return true. If it is well-formed, return false. */ static int check_name(const char *z){ if( !wiki_name_is_wellformed((const unsigned char *)z) ){ style_header("Wiki Page Name Error"); - @ The wiki name "<b>%h(z)</b>" is not well-formed. Rules for - @ wiki page names: + @ The wiki name "<span class="wikiError">%h(z)</span>" is not well-formed. + @ Rules for wiki page names: well_formed_wiki_name_rules(); style_footer(); return 1; } return 0; @@ -79,14 +79,23 @@ ** WEBPAGE: index ** WEBPAGE: not_found */ void home_page(void){ char *zPageName = db_get("project-name",0); + char *zIndexPage = db_get("index-page",0); login_check_credentials(); if( !g.okRdWiki ){ cgi_redirectf("%s/login?g=%s/home", g.zBaseURL, g.zBaseURL); } + if( zIndexPage ){ + const char *zPathInfo = P("PATH_INFO"); + while( zIndexPage[0]=='/' ) zIndexPage++; + if( strcmp(zIndexPage, zPathInfo)==0 ) zIndexPage = 0; + } + if( zIndexPage ){ + cgi_redirectf("%s/%s", g.zBaseURL, zIndexPage); + } if( zPageName ){ login_check_credentials(); g.zExtra = zPageName; cgi_set_parameter_nocopy("name", g.zExtra); g.isHome = 1; @@ -97,11 +106,11 @@ @ <p>This is a stub home-page for the project. @ To fill in this page, first go to @ <a href="%s(g.zBaseURL)/setup_config">setup/config</a> @ and establish a "Project Name". Then create a @ wiki page with that name. The content of that wiki page - @ will be displayed in place of this message. + @ will be displayed in place of this message.</p> style_footer(); } /* ** Return true if the given pagename is the name of the sandbox @@ -118,11 +127,11 @@ void wiki_page(void){ char *zTag; int rid = 0; int isSandbox; Blob wiki; - Manifest m; + Manifest *pWiki = 0; const char *zPageName; char *zBody = mprintf("%s","<i>Empty Page</i>"); Stmt q; int cnt = 0; @@ -144,17 +153,20 @@ @ wiki.</li> @ <li> Use the <a href="%s(g.zBaseURL)/wiki?name=Sandbox">Sandbox</a> @ to experiment.</li> if( g.okNewWiki ){ @ <li> Create a <a href="%s(g.zBaseURL)/wikinew">new wiki page</a>.</li> + if( g.okWrite ){ + @ <li> Create a <a href="%s(g.zTop)/eventedit">new event</a>.</li> + } } @ <li> <a href="%s(g.zBaseURL)/wcontent">List of All Wiki Pages</a> @ available on this server.</li> - @ <li> <form method="GET" action="%s(g.zBaseURL)/wfind"> - @ Search wiki titles: <input type="text" name="title"/> - @ <input type="submit" /> - @ </li> + @ <li> <form method="get" action="%s(g.zBaseURL)/wfind"><div> + @ Search wiki titles: <input type="text" name="title"/> + @ <input type="submit" /></div></form> + @ </li> @ </ul> style_footer(); return; } if( check_name(zPageName) ) return; @@ -167,30 +179,24 @@ "SELECT rid FROM tagxref" " WHERE tagid=(SELECT tagid FROM tag WHERE tagname=%Q)" " ORDER BY mtime DESC", zTag ); free(zTag); - memset(&m, 0, sizeof(m)); - blob_zero(&m.content); - if( rid ){ - Blob content; - content_get(rid, &content); - manifest_parse(&m, &content); - if( m.type==CFTYPE_WIKI && m.zWiki ){ - while( isspace(m.zWiki[0]) ) m.zWiki++; - if( m.zWiki[0] ) zBody = m.zWiki; - } + pWiki = manifest_get(rid, CFTYPE_WIKI); + if( pWiki ){ + while( fossil_isspace(pWiki->zWiki[0]) ) pWiki->zWiki++; + if( pWiki->zWiki[0] ) zBody = pWiki->zWiki; } } if( !g.isHome ){ if( (rid && g.okWrWiki) || (!rid && g.okNewWiki) ){ style_submenu_element("Edit", "Edit Wiki Page", "%s/wikiedit?name=%T", g.zTop, zPageName); } if( rid && g.okApndWiki && g.okAttach ){ style_submenu_element("Attach", "Add An Attachment", - "%s/attachadd?page=%T&from=%s/wiki%%3fname=%T", + "%s/attachadd?page=%T&from=%s/wiki%%3fname=%T", g.zTop, zPageName, g.zTop, zPageName); } if( rid && g.okApndWiki ){ style_submenu_element("Append", "Add A Comment", "%s/wikiappend?name=%T", g.zTop, zPageName); @@ -214,33 +220,34 @@ while( db_step(&q)==SQLITE_ROW ){ const char *zDate = db_column_text(&q, 0); const char *zFile = db_column_text(&q, 1); const char *zUser = db_column_text(&q, 2); if( cnt==0 ){ - @ <hr><h2>Attachments:</h2> + @ <hr /><h2>Attachments:</h2> @ <ul> } cnt++; - if( g.okHistory ){ - @ <li><a href="%s(g.zTop)/attachview?page=%s(zPageName)&file=%t(zFile)"> + @ <li> + if( g.okHistory && g.okRead ){ + @ <a href="%s(g.zTop)/attachview?page=%s(zPageName)&file=%t(zFile)"> + @ %h(zFile)</a> }else{ - @ <li> + @ <li>%h(zFile) } - @ %h(zFile)</a> add by %h(zUser) on + @ added by %h(zUser) on hyperlink_to_date(zDate, "."); if( g.okWrWiki && g.okAttach ){ - @ [<a href="%s(g.zTop)/attachdelete?page=%s(zPageName)&file=%t(zFile)&from=%s(g.zTop)/wiki%%3fname=%s(zPageName)">delete</a>] + @ [<a href="%s(g.zTop)/attachdelete?page=%s(zPageName)&file=%t(zFile)&from=%s(g.zTop)/wiki%%3fname=%s(zPageName)">delete</a>] } + @ </li> } if( cnt ){ @ </ul> } db_finalize(&q); - if( !isSandbox ){ - manifest_clear(&m); - } + manifest_destroy(pWiki); style_footer(); } /* ** WEBPAGE: wikiedit @@ -249,11 +256,11 @@ void wikiedit_page(void){ char *zTag; int rid = 0; int isSandbox; Blob wiki; - Manifest m; + Manifest *pWiki = 0; const char *zPageName; char *zHtmlPageName; int n; const char *z; char *zBody = (char*)P("w"); @@ -283,19 +290,12 @@ free(zTag); if( (rid && !g.okWrWiki) || (!rid && !g.okNewWiki) ){ login_needed(); return; } - memset(&m, 0, sizeof(m)); - blob_zero(&m.content); - if( rid && zBody==0 ){ - Blob content; - content_get(rid, &content); - manifest_parse(&m, &content); - if( m.type==CFTYPE_WIKI ){ - zBody = m.zWiki; - } + if( zBody==0 && (pWiki = manifest_get(rid, CFTYPE_WIKI))!=0 ){ + zBody = pWiki->zWiki; } } if( P("submit")!=0 && zBody!=0 ){ char *zDate; Blob cksum; @@ -342,33 +342,31 @@ zHtmlPageName = mprintf("Edit: %s", zPageName); style_header(zHtmlPageName); if( P("preview")!=0 ){ blob_zero(&wiki); blob_append(&wiki, zBody, -1); - @ Preview:<hr> + @ Preview:<hr /> wiki_convert(&wiki, 0, 0); - @ <hr> + @ <hr /> blob_reset(&wiki); } for(n=2, z=zBody; z[0]; z++){ if( z[0]=='\n' ) n++; } if( n<20 ) n = 20; if( n>40 ) n = 40; - @ <form method="POST" action="%s(g.zBaseURL)/wikiedit"> + @ <form method="post" action="%s(g.zBaseURL)/wikiedit"><div> login_insert_csrf_secret(); - @ <input type="hidden" name="name" value="%h(zPageName)"> + @ <input type="hidden" name="name" value="%h(zPageName)" /> @ <textarea name="w" class="wikiedit" cols="80" @ rows="%d(n)" wrap="virtual">%h(zBody)</textarea> - @ <br> - @ <input type="submit" name="preview" value="Preview Your Changes"> - @ <input type="submit" name="submit" value="Apply These Changes"> - @ <input type="submit" name="cancel" value="Cancel"> - @ </form> - if( !isSandbox ){ - manifest_clear(&m); - } + @ <br /> + @ <input type="submit" name="preview" value="Preview Your Changes" /> + @ <input type="submit" name="submit" value="Apply These Changes" /> + @ <input type="submit" name="cancel" value="Cancel" /> + @ </div></form> + manifest_destroy(pWiki); style_footer(); } /* ** WEBPAGE: wikinew @@ -387,21 +385,20 @@ zName = PD("name",""); if( zName[0] && wiki_name_is_wellformed((const unsigned char *)zName) ){ cgi_redirectf("wikiedit?name=%T", zName); } style_header("Create A New Wiki Page"); - @ <p>Rules for wiki page names: + @ <p>Rules for wiki page names:</p> well_formed_wiki_name_rules(); - @ </p> - @ <form method="POST" action="%s(g.zBaseURL)/wikinew"> + @ <form method="post" action="%s(g.zBaseURL)/wikinew"> @ <p>Name of new wiki page: - @ <input type="text" width="35" name="name" value="%h(zName)"> - @ <input type="submit" value="Create"> + @ <input style="width: 35;" type="text" name="name" value="%h(zName)" /> + @ <input type="submit" value="Create" /> @ </p></form> if( zName[0] ){ - @ <p><b><font color="red"> - @ "%h(zName)" is not a valid wiki page name!</font></b></p> + @ <p><span class="wikiError"> + @ "%h(zName)" is not a valid wiki page name!</span></p> } style_footer(); } @@ -463,27 +460,25 @@ if( P("submit")!=0 && P("r")!=0 && P("u")!=0 ){ char *zDate; Blob cksum; int nrid; Blob body; - Blob content; Blob wiki; - Manifest m; + Manifest *pWiki = 0; blob_zero(&body); if( isSandbox ){ blob_appendf(&body, db_get("sandbox","")); appendRemark(&body); db_set("sandbox", blob_str(&body), 0); }else{ login_verify_csrf_secret(); - content_get(rid, &content); - manifest_parse(&m, &content); - if( m.type==CFTYPE_WIKI ){ - blob_append(&body, m.zWiki, -1); + pWiki = manifest_get(rid, CFTYPE_WIKI); + if( pWiki ){ + blob_append(&body, pWiki->zWiki, -1); + manifest_destroy(pWiki); } - manifest_clear(&m); blob_zero(&wiki); db_begin_transaction(); zDate = db_text(0, "SELECT datetime('now')"); zDate[10] = 'T'; blob_appendf(&wiki, "D %s\n", zDate); @@ -524,22 +519,22 @@ wiki_convert(&preview, 0, 0); @ <hr> blob_reset(&preview); } zUser = PD("u", g.zLogin); - @ <form method="POST" action="%s(g.zBaseURL)/wikiappend"> + @ <form method="post" action="%s(g.zBaseURL)/wikiappend"> login_insert_csrf_secret(); - @ <input type="hidden" name="name" value="%h(zPageName)"> + @ <input type="hidden" name="name" value="%h(zPageName)" /> @ Your Name: - @ <input type="text" name="u" size="20" value="%h(zUser)"><br> - @ Comment to append:<br> + @ <input type="text" name="u" size="20" value="%h(zUser)" /><br /> + @ Comment to append:<br /> @ <textarea name="r" class="wikiedit" cols="80" @ rows="10" wrap="virtual">%h(PD("r",""))</textarea> - @ <br> - @ <input type="submit" name="preview" value="Preview Your Comment"> - @ <input type="submit" name="submit" value="Append Your Changes"> - @ <input type="submit" name="cancel" value="Cancel"> + @ <br /> + @ <input type="submit" name="preview" value="Preview Your Comment" /> + @ <input type="submit" name="submit" value="Append Your Changes" /> + @ <input type="submit" name="cancel" value="Cancel" /> @ </form> style_footer(); } /* @@ -551,11 +546,11 @@ ** Function called to output extra text at the end of each line in ** a wiki history listing. */ static void wiki_history_extra(int rid){ if( db_exists("SELECT 1 FROM tagxref WHERE rid=%d", rid) ){ - @ <a href="%s(g.zTop)/wdiff?name=%t(zWikiPageName)&a=%d(rid)">[diff]</a> + @ <a href="%s(g.zTop)/wdiff?name=%t(zWikiPageName)&a=%d(rid)">[diff]</a> } } /* ** WEBPAGE: whistory @@ -598,12 +593,11 @@ */ void wdiff_page(void){ char *zTitle; int rid1, rid2; const char *zPageName; - Blob content1, content2; - Manifest m1, m2; + Manifest *pW1, *pW2 = 0; Blob w1, w2, d; login_check_credentials(); rid1 = atoi(PD("a","0")); if( !g.okHistory ){ login_needed(); return; } @@ -621,27 +615,24 @@ " WHERE event.mtime<(SELECT mtime FROM event WHERE objid=%d)" " ORDER BY event.mtime DESC LIMIT 1", zPageName, rid1 ); } - content_get(rid1, &content1); - manifest_parse(&m1, &content1); - if( m1.type!=CFTYPE_WIKI ) fossil_redirect_home(); - blob_init(&w1, m1.zWiki, -1); + pW1 = manifest_get(rid1, CFTYPE_WIKI); + if( pW1==0 ) fossil_redirect_home(); + blob_init(&w1, pW1->zWiki, -1); blob_zero(&w2); - if( rid2 ){ - content_get(rid2, &content2); - manifest_parse(&m2, &content2); - if( m2.type==CFTYPE_WIKI ){ - blob_init(&w2, m2.zWiki, -1); - } + if( rid2 && (pW2 = manifest_get(rid2, CFTYPE_WIKI))!=0 ){ + blob_init(&w2, pW2->zWiki, -1); } blob_zero(&d); - text_diff(&w2, &w1, &d, 5); + text_diff(&w2, &w1, &d, 5, 1); @ <pre> @ %h(blob_str(&d)) @ </pre> + manifest_destroy(pW1); + manifest_destroy(pW2); style_footer(); } /* ** WEBPAGE: wcontent @@ -731,98 +722,53 @@ @ </ol> @ <p>We call the first five rules above "wiki" formatting rules. The @ last two rules are the HTML formatting rule.</p> @ <h2>Formatting Rule Details</h2> @ <ol> - @ <li> <p><b>Paragraphs</b>. Any sequence of one or more blank lines forms + @ <li> <p><span class="wikiruleHead">Paragraphs</span>. Any sequence of one or more blank lines forms @ a paragraph break. Centered or right-justified paragraphs are not @ supported by wiki markup, but you can do these things if you need them - @ using HTML.</p> - @ <li> <p><b>Bullet Lists</b>. + @ using HTML.</p></li> + @ <li> <p><span class="wikiruleHead">Bullet Lists</span>. @ A bullet list item is a line that begins with a single "*" character @ surrounded on @ both sides by two or more spaces or by a tab. Only a single level - @ of bullet list is supported by wiki. For nested lists, use HTML.</p> - @ <li> <p><b>Enumeration Lists</b>. + @ of bullet list is supported by wiki. For nested lists, use HTML.</p></li> + @ <li> <p><span class="wikiruleHead">Enumeration Lists</span>. @ An enumeration list item is a line that begins with a single "#" character @ surrounded on both sides by two or more spaces or by a tab. Only a single @ level of enumeration list is supported by wiki. For nested lists or for - @ enumerations that count using letters or roman numerials, use HTML.</p> - @ <li> <p><b>Indented Paragraphs</b>. + @ enumerations that count using letters or roman numerials, use HTML.</p></li> + @ <li> <p><span class="wikiruleHead">Indented Paragraphs</span>. @ Any paragraph that begins with two or more spaces or a tab and @ which is not a bullet or enumeration list item is rendered @ indented. Only a single level of indentation is supported by wiki; use - @ HTML for deeper indentation.</p> - @ <li> <p><b>Hyperlinks</b>. + @ HTML for deeper indentation.</p></li> + @ <li> <p><span class="wikiruleHead">Hyperlinks</span>. @ Text within square brackets ("[...]") becomes a hyperlink. The @ target can be a wiki page name, the artifact ID of a check-in or ticket, @ the name of an image, or a URL. By default, the target is displayed @ as the text of the hyperlink. But you can specify alternative text @ after the target name separated by a "|" character.</p> @ <p>You can also link to internal anchor names using [#anchor-name], providing @ you have added the necessary "<a name="anchor-name"></a>" - @ tag to your wiki page.</p> - @ <li> <p><b>HTML</b>. + @ tag to your wiki page.</p></li> + @ <li> <p><span class="wikiruleHead">HTML</span>. @ The following standard HTML elements may be used: - @ <a> - @ <address> - @ <b> - @ <big> - @ <blockquote> - @ <br> - @ <center> - @ <cite> - @ <code> - @ <dd> - @ <dfn> - @ <div> - @ <dl> - @ <dt> - @ <em> - @ <font> - @ <h1> - @ <h2> - @ <h3> - @ <h4> - @ <h5> - @ <h6> - @ <hr> - @ <img> - @ <i> - @ <kbd> - @ <li> - @ <nobr> - @ <ol> - @ <p> - @ <pre> - @ <s> - @ <samp> - @ <small> - @ <strike> - @ <strong> - @ <sub> - @ <sup> - @ <table> - @ <td> - @ <th> - @ <tr> - @ <tt> - @ <u> - @ <ul> - @ <var>. - @ In addition, there are two non-standard elements available: + show_allowed_wiki_markup(); + @ . There are two non-standard elements available: @ <verbatim> and <nowiki>. @ No other elements are allowed. All attributes are checked and @ only a few benign attributes are allowed on each element. @ In particular, any attributes that specify javascript or CSS @ are elided.</p></li> - @ <li><p><b>Special Markup.</b> + @ <li><p><span class="wikiruleHead">Special Markup.</span> @ The <nowiki> tag disables all wiki formatting rules @ through the matching </nowiki> element. @ The <verbatim> tag works like <pre> with the addition @ that it also disables all wiki and HTML markup - @ through the matching </verbatim>. + @ through the matching </verbatim>.</p></li> @ </ol> style_footer(); } /* @@ -945,35 +891,31 @@ } if( strncmp(g.argv[2],"export",n)==0 ){ char const *zPageName; /* Name of the wiki page to export */ char const *zFile; /* Name of the output file (0=stdout) */ - int rid; /* Artifact ID of the wiki page */ - int i; /* Loop counter */ - char *zBody = 0; /* Wiki page content */ - Manifest m; /* Parsed wiki page content */ + int rid; /* Artifact ID of the wiki page */ + int i; /* Loop counter */ + char *zBody = 0; /* Wiki page content */ + Manifest *pWiki = 0; /* Parsed wiki page content */ + if( (g.argc!=4) && (g.argc!=5) ){ usage("export PAGENAME ?FILE?"); } zPageName = g.argv[3]; rid = db_int(0, "SELECT x.rid FROM tag t, tagxref x" " WHERE x.tagid=t.tagid AND t.tagname='wiki-%q'" " ORDER BY x.mtime DESC LIMIT 1", zPageName ); - if( rid ){ - Blob content; - content_get(rid, &content); - manifest_parse(&m, &content); - if( m.type==CFTYPE_WIKI ){ - zBody = m.zWiki; - } + if( (pWiki = manifest_get(rid, CFTYPE_WIKI))!=0 ){ + zBody = pWiki->zWiki; } if( zBody==0 ){ fossil_fatal("wiki page [%s] not found",zPageName); } - for(i=strlen(zBody); i>0 && isspace(zBody[i-1]); i--){} + for(i=strlen(zBody); i>0 && fossil_isspace(zBody[i-1]); i--){} zFile = (g.argc==4) ? 0 : g.argv[4]; if( zFile ){ FILE * zF; short doClose = 0; if( (1 == strlen(zFile)) && ('-'==zFile[0]) ){ @@ -986,12 +928,13 @@ fossil_fatal("wiki export could not open output file for writing."); } fprintf(zF,"%.*s\n", i, zBody); if( doClose ) fclose(zF); }else{ - printf("%.*s\n", i, zBody); + printf("%.*s\n", i, zBody); } + manifest_destroy(pWiki); return; }else if( strncmp(g.argv[2],"commit",n)==0 || strncmp(g.argv[2],"create",n)==0 ){ char *zPageName; Index: src/wikiformat.c ================================================================== --- src/wikiformat.c +++ src/wikiformat.c @@ -150,59 +150,65 @@ ** Except for MARKUP_INVALID, this must all be in alphabetical order ** and in numerical sequence. The first markup type must be zero. ** The value for MARKUP_XYZ must correspond to the <xyz> entry ** in aAllowedMarkup[]. */ -#define MARKUP_INVALID 0 -#define MARKUP_A 1 -#define MARKUP_ADDRESS 2 -#define MARKUP_B 3 -#define MARKUP_BIG 4 -#define MARKUP_BLOCKQUOTE 5 -#define MARKUP_BR 6 -#define MARKUP_CENTER 7 -#define MARKUP_CITE 8 -#define MARKUP_CODE 9 -#define MARKUP_DD 10 -#define MARKUP_DFN 11 -#define MARKUP_DIV 12 -#define MARKUP_DL 13 -#define MARKUP_DT 14 -#define MARKUP_EM 15 -#define MARKUP_FONT 16 -#define MARKUP_H1 17 -#define MARKUP_H2 18 -#define MARKUP_H3 19 -#define MARKUP_H4 20 -#define MARKUP_H5 21 -#define MARKUP_H6 22 -#define MARKUP_HR 23 -#define MARKUP_I 24 -#define MARKUP_IMG 25 -#define MARKUP_KBD 26 -#define MARKUP_LI 27 -#define MARKUP_NOBR 28 -#define MARKUP_NOWIKI 29 -#define MARKUP_OL 30 -#define MARKUP_P 31 -#define MARKUP_PRE 32 -#define MARKUP_S 33 -#define MARKUP_SAMP 34 -#define MARKUP_SMALL 35 -#define MARKUP_STRIKE 36 -#define MARKUP_STRONG 37 -#define MARKUP_SUB 38 -#define MARKUP_SUP 39 -#define MARKUP_TABLE 40 -#define MARKUP_TD 41 -#define MARKUP_TH 42 -#define MARKUP_TR 43 -#define MARKUP_TT 44 -#define MARKUP_U 45 -#define MARKUP_UL 46 -#define MARKUP_VAR 47 -#define MARKUP_VERBATIM 48 +#define MARKUP_INVALID 0 +#define MARKUP_A 1 +#define MARKUP_ADDRESS 2 +#define MARKUP_B 3 +#define MARKUP_BIG 4 +#define MARKUP_BLOCKQUOTE 5 +#define MARKUP_BR 6 +#define MARKUP_CENTER 7 +#define MARKUP_CITE 8 +#define MARKUP_CODE 9 +#define MARKUP_COL 10 +#define MARKUP_COLGROUP 11 +#define MARKUP_DD 12 +#define MARKUP_DFN 13 +#define MARKUP_DIV 14 +#define MARKUP_DL 15 +#define MARKUP_DT 16 +#define MARKUP_EM 17 +#define MARKUP_FONT 18 +#define MARKUP_H1 19 +#define MARKUP_H2 20 +#define MARKUP_H3 21 +#define MARKUP_H4 22 +#define MARKUP_H5 23 +#define MARKUP_H6 24 +#define MARKUP_HR 25 +#define MARKUP_I 26 +#define MARKUP_IMG 27 +#define MARKUP_KBD 28 +#define MARKUP_LI 29 +#define MARKUP_NOBR 30 +#define MARKUP_NOWIKI 31 +#define MARKUP_OL 32 +#define MARKUP_P 33 +#define MARKUP_PRE 34 +#define MARKUP_S 35 +#define MARKUP_SAMP 36 +#define MARKUP_SMALL 37 +#define MARKUP_SPAN 38 +#define MARKUP_STRIKE 39 +#define MARKUP_STRONG 40 +#define MARKUP_SUB 41 +#define MARKUP_SUP 42 +#define MARKUP_TABLE 43 +#define MARKUP_TBODY 44 +#define MARKUP_TD 45 +#define MARKUP_TFOOT 46 +#define MARKUP_TH 47 +#define MARKUP_THEAD 48 +#define MARKUP_TR 49 +#define MARKUP_TT 50 +#define MARKUP_U 51 +#define MARKUP_UL 52 +#define MARKUP_VAR 53 +#define MARKUP_VERBATIM 54 /* ** The various markup is divided into the following types: */ #define MUTYPE_SINGLE 0x0001 /* <img>, <br>, or <hr> */ @@ -241,10 +247,14 @@ { "blockquote", MARKUP_BLOCKQUOTE, MUTYPE_BLOCK, 0 }, { "br", MARKUP_BR, MUTYPE_SINGLE, AMSK_CLEAR }, { "center", MARKUP_CENTER, MUTYPE_BLOCK, 0 }, { "cite", MARKUP_CITE, MUTYPE_FONT, 0 }, { "code", MARKUP_CODE, MUTYPE_FONT, 0 }, + { "col", MARKUP_COL, MUTYPE_SINGLE, + AMSK_ALIGN|AMSK_CLASS|AMSK_COLSPAN|AMSK_WIDTH }, + { "colgroup", MARKUP_COLGROUP, MUTYPE_BLOCK, + AMSK_ALIGN|AMSK_CLASS|AMSK_COLSPAN|AMSK_WIDTH}, { "dd", MARKUP_DD, MUTYPE_LI, 0 }, { "dfn", MARKUP_DFN, MUTYPE_FONT, 0 }, { "div", MARKUP_DIV, MUTYPE_BLOCK, AMSK_ID|AMSK_CLASS }, { "dl", MARKUP_DL, MUTYPE_LIST, AMSK_COMPACT }, { "dt", MARKUP_DT, MUTYPE_LI, 0 }, @@ -273,32 +283,44 @@ { "p", MARKUP_P, MUTYPE_BLOCK, AMSK_ALIGN|AMSK_CLASS }, { "pre", MARKUP_PRE, MUTYPE_BLOCK, 0 }, { "s", MARKUP_S, MUTYPE_FONT, 0 }, { "samp", MARKUP_SAMP, MUTYPE_FONT, 0 }, { "small", MARKUP_SMALL, MUTYPE_FONT, 0 }, + { "span", MARKUP_SPAN, MUTYPE_BLOCK, AMSK_ALIGN|AMSK_CLASS }, { "strike", MARKUP_STRIKE, MUTYPE_FONT, 0 }, { "strong", MARKUP_STRONG, MUTYPE_FONT, 0 }, { "sub", MARKUP_SUB, MUTYPE_FONT, 0 }, { "sup", MARKUP_SUP, MUTYPE_FONT, 0 }, { "table", MARKUP_TABLE, MUTYPE_TABLE, AMSK_ALIGN|AMSK_BGCOLOR|AMSK_BORDER|AMSK_CELLPADDING| AMSK_CELLSPACING|AMSK_HSPACE|AMSK_VSPACE|AMSK_CLASS }, + { "tbody", MARKUP_TBODY, MUTYPE_BLOCK, AMSK_ALIGN|AMSK_CLASS }, { "td", MARKUP_TD, MUTYPE_TD, AMSK_ALIGN|AMSK_BGCOLOR|AMSK_COLSPAN| AMSK_ROWSPAN|AMSK_VALIGN|AMSK_CLASS }, + { "tfoot", MARKUP_TFOOT, MUTYPE_BLOCK, AMSK_ALIGN|AMSK_CLASS }, { "th", MARKUP_TH, MUTYPE_TD, AMSK_ALIGN|AMSK_BGCOLOR|AMSK_COLSPAN| AMSK_ROWSPAN|AMSK_VALIGN|AMSK_CLASS }, + { "thead", MARKUP_THEAD, MUTYPE_BLOCK, AMSK_ALIGN|AMSK_CLASS }, { "tr", MARKUP_TR, MUTYPE_TR, AMSK_ALIGN|AMSK_BGCOLOR||AMSK_VALIGN|AMSK_CLASS }, { "tt", MARKUP_TT, MUTYPE_FONT, 0 }, { "u", MARKUP_U, MUTYPE_FONT, 0 }, { "ul", MARKUP_UL, MUTYPE_LIST, AMSK_TYPE|AMSK_COMPACT }, { "var", MARKUP_VAR, MUTYPE_FONT, 0 }, { "verbatim", MARKUP_VERBATIM, MUTYPE_SPECIAL, AMSK_ID|AMSK_TYPE }, }; + +void show_allowed_wiki_markup( void ){ + int i; /* loop over allowedAttr */ + + for( i=1 ; i<=sizeof(aMarkup)/sizeof(aMarkup[0]) - 1 ; i++ ){ + @ <%s(aMarkup[i].zName)> + } +} /* ** Use binary search to locate a tag in the aMarkup[] table. */ static int findTag(const char *z){ @@ -321,21 +343,21 @@ } /* ** Token types */ -#define TOKEN_MARKUP 1 /* <...> */ -#define TOKEN_CHARACTER 2 /* "&" or "<" not part of markup */ -#define TOKEN_LINK 3 /* [...] */ -#define TOKEN_PARAGRAPH 4 /* blank lines */ -#define TOKEN_NEWLINE 5 /* A single "\n" */ -#define TOKEN_BUL_LI 6 /* " * " */ -#define TOKEN_NUM_LI 7 /* " # " */ -#define TOKEN_ENUM 8 /* " \(?\d+[.)]? " */ -#define TOKEN_INDENT 9 /* " " */ -#define TOKEN_RAW 10 /* Output exactly (used when wiki-use-html==1) */ -#define TOKEN_TEXT 11 /* None of the above */ +#define TOKEN_MARKUP 1 /* <...> */ +#define TOKEN_CHARACTER 2 /* "&" or "<" not part of markup */ +#define TOKEN_LINK 3 /* [...] */ +#define TOKEN_PARAGRAPH 4 /* blank lines */ +#define TOKEN_NEWLINE 5 /* A single "\n" */ +#define TOKEN_BUL_LI 6 /* " * " */ +#define TOKEN_NUM_LI 7 /* " # " */ +#define TOKEN_ENUM 8 /* " \(?\d+[.)]? " */ +#define TOKEN_INDENT 9 /* " " */ +#define TOKEN_RAW 10 /* Output exactly (used when wiki-use-html==1) */ +#define TOKEN_TEXT 11 /* None of the above */ /* ** State flags */ #define AT_NEWLINE 0x001 /* At start of a line */ @@ -379,11 +401,10 @@ static int r = -1; if( r<0 ) r = db_get_boolean("wiki-use-html", 0); return r; } - /* ** z points to a "<" character. Check to see if this is the start of ** a valid markup. If it is, return the total number of characters in ** the markup including the initial "<" and the terminating ">". If ** it is not well-formed markup, return 0. @@ -391,17 +412,19 @@ static int markupLength(const char *z){ int n = 1; int inparen = 0; int c; if( z[n]=='/' ){ n++; } - if( !isalpha(z[n]) ) return 0; - while( isalnum(z[n]) ){ n++; } - if( (c = z[n])!='>' && !isspace(c) ) return 0; + if( !fossil_isalpha(z[n]) ) return 0; + while( fossil_isalnum(z[n]) ){ n++; } + c = z[n]; + if( c=='/' && z[n+1]=='>' ){ return n+2; } + if( c!='>' && !fossil_isspace(c) ) return 0; while( (c = z[n])!=0 && (c!='>' || inparen) ){ if( c==inparen ){ inparen = 0; - }else if( c=='"' || c=='\'' ){ + }else if( inparen==0 && (c=='"' || c=='\'') ){ inparen = c; } n++; } if( z[n]!='>' ) return 0; @@ -414,11 +437,11 @@ ** of characters through the closing "\n". If not, return 0. */ static int paragraphBreakLength(const char *z){ int i, n; int nNewline = 1; - for(i=1, n=0; isspace(z[i]); i++){ + for(i=1, n=0; fossil_isspace(z[i]); i++){ if( z[i]=='\n' ){ nNewline++; n = i; } } @@ -459,14 +482,14 @@ */ static int isElement(const char *z){ int i; assert( z[0]=='&' ); if( z[1]=='#' ){ - for(i=2; isdigit(z[i]); i++){} + for(i=2; fossil_isdigit(z[i]); i++){} return i>2 && z[i]==';'; }else{ - for(i=1; isalpha(z[i]); i++){} + for(i=1; fossil_isalpha(z[i]); i++){} return i>1 && z[i]==';'; } } /* @@ -488,11 +511,11 @@ while( z[n]==' ' || z[n]=='\t' ){ if( z[n]=='\t' ) i++; i++; n++; } - if( i<2 || isspace(z[n]) ) return 0; + if( i<2 || fossil_isspace(z[n]) ) return 0; return n; } /* ** Check to see if the z[] string is the beginning of a enumeration value. @@ -513,11 +536,11 @@ if( z[n]=='\t' ) i++; i++; n++; } if( i<2 ) return 0; - for(i=0; isdigit(z[n]); i++, n++){} + for(i=0; fossil_isdigit(z[n]); i++, n++){} if( i==0 ) return 0; if( z[n]=='.' ){ n++; } i = 0; @@ -524,11 +547,11 @@ while( z[n]==' ' || z[n]=='\t' ){ if( z[n]=='\t' ) i++; i++; n++; } - if( i<2 || isspace(z[n]) ) return 0; + if( i<2 || fossil_isspace(z[n]) ) return 0; return n; } /* ** Check to see if the z[] string is the beginning of an indented @@ -542,11 +565,11 @@ while( z[n]==' ' || z[n]=='\t' ){ if( z[n]=='\t' ) i++; i++; n++; } - if( i<2 || isspace(z[n]) ) return 0; + if( i<2 || fossil_isspace(z[n]) ) return 0; return n; } /* ** Check to see if the z[] string is a wiki hyperlink. If it is, @@ -589,16 +612,16 @@ if( z[0]=='\n' ){ n = paragraphBreakLength(z); if( n>0 ){ *pTokenType = TOKEN_PARAGRAPH; return n; - }else if( isspace(z[1]) ){ + }else if( fossil_isspace(z[1]) ){ *pTokenType = TOKEN_NEWLINE; return 1; } } - if( (p->state & AT_NEWLINE)!=0 && isspace(z[0]) ){ + if( (p->state & AT_NEWLINE)!=0 && fossil_isspace(z[0]) ){ n = listItemLength(z, '*'); if( n>0 ){ *pTokenType = TOKEN_BUL_LI; return n; } @@ -611,11 +634,11 @@ if( n>0 ){ *pTokenType = TOKEN_ENUM; return n; } } - if( (p->state & AT_PARAGRAPH)!=0 && isspace(z[0]) ){ + if( (p->state & AT_PARAGRAPH)!=0 && fossil_isspace(z[0]) ){ n = indentLength(z); if( n>0 ){ *pTokenType = TOKEN_INDENT; return n; } @@ -682,37 +705,37 @@ }else{ p->endTag = 0; i = 1; } j = 0; - while( isalnum(z[i]) ){ - if( j<sizeof(zTag)-1 ) zTag[j++] = tolower(z[i]); + while( fossil_isalnum(z[i]) ){ + if( j<sizeof(zTag)-1 ) zTag[j++] = fossil_tolower(z[i]); i++; } zTag[j] = 0; p->iCode = findTag(zTag); p->iType = aMarkup[p->iCode].iType; p->nAttr = 0; - while( isspace(z[i]) ){ i++; } - while( p->nAttr<8 && isalpha(z[i]) ){ + while( fossil_isspace(z[i]) ){ i++; } + while( p->nAttr<8 && fossil_isalpha(z[i]) ){ int attrOk; /* True to preserver attribute. False to ignore it */ j = 0; - while( isalnum(z[i]) ){ - if( j<sizeof(zTag)-1 ) zTag[j++] = tolower(z[i]); + while( fossil_isalnum(z[i]) ){ + if( j<sizeof(zTag)-1 ) zTag[j++] = fossil_tolower(z[i]); i++; } zTag[j] = 0; p->aAttr[p->nAttr].iACode = iACode = findAttr(zTag); attrOk = iACode!=0 && (seen & aAttribute[iACode].iMask)==0; - while( isspace(z[i]) ){ z++; } + while( fossil_isspace(z[i]) ){ z++; } if( z[i]!='=' ){ p->aAttr[p->nAttr].zValue = 0; p->aAttr[p->nAttr].cTerm = 0; c = 0; }else{ i++; - while( isspace(z[i]) ){ z++; } + while( fossil_isspace(z[i]) ){ z++; } if( z[i]=='"' ){ i++; zValue = &z[i]; while( z[i] && z[i]!='"' ){ i++; } }else if( z[i]=='\'' ){ @@ -719,11 +742,11 @@ i++; zValue = &z[i]; while( z[i] && z[i]!='\'' ){ i++; } }else{ zValue = &z[i]; - while( !isspace(z[i]) && z[i]!='>' ){ z++; } + while( !fossil_isspace(z[i]) && z[i]!='>' ){ z++; } } if( attrOk ){ p->aAttr[p->nAttr].zValue = zValue; p->aAttr[p->nAttr].cTerm = c = z[i]; z[i] = 0; @@ -732,11 +755,11 @@ } if( attrOk ){ seen |= aAttribute[iACode].iMask; p->nAttr++; } - while( isspace(z[i]) ){ i++; } + while( fossil_isspace(z[i]) ){ i++; } if( z[i]=='>' || (z[i]=='/' && z[i+1]=='>') ) break; } } /* @@ -756,10 +779,13 @@ blob_appendf(pOut, "=\"%s%s\"", g.zBaseURL, zVal); }else{ blob_appendf(pOut, "=\"%s\"", zVal); } } + } + if (p->iType & MUTYPE_SINGLE){ + blob_append(pOut, " /", 2); } blob_append(pOut, ">", 1); } } @@ -812,14 +838,11 @@ ** if necessary. */ static void pushStackWithId(Renderer *p, int elem, const char *zId, int w){ if( p->nStack>=p->nAlloc ){ p->nAlloc = p->nAlloc*2 + 100; - p->aStack = realloc(p->aStack, p->nAlloc*sizeof(p->aStack[0])); - if( p->aStack==0 ){ - fossil_panic("out of memory"); - } + p->aStack = fossil_realloc(p->aStack, p->nAlloc*sizeof(p->aStack[0])); } p->aStack[p->nStack].iCode = elem; p->aStack[p->nStack].zId = zId; p->aStack[p->nStack].allowWiki = w; p->nStack++; @@ -885,11 +908,12 @@ /* ** Begin a new paragraph if that something that is needed. */ static void startAutoParagraph(Renderer *p){ - if( p->wantAutoParagraph==0 || p->wikiList==MARKUP_OL || p->wikiList==MARKUP_UL ) return; + if( p->wantAutoParagraph==0 ) return; + if( p->wikiList==MARKUP_OL || p->wikiList==MARKUP_UL ) return; blob_appendf(p->pOut, "<p>", -1); pushStack(p, MARKUP_P); p->wantAutoParagraph = 0; p->inAutoParagraph = 1; } @@ -1019,17 +1043,18 @@ /* Special display processing for tickets. Display the hyperlink ** as crossed out if the ticket is closed. */ if( isClosed ){ if( g.okHistory ){ - blob_appendf(p->pOut,"<a href=\"%s/info/%s\"><s>", - g.zBaseURL, zTarget + blob_appendf(p->pOut, + "<a href=\"%s/info/%s\"><span class=\"wikiTagCancelled\">", + g.zBaseURL, zTarget ); - zTerm = "</s></a>"; + zTerm = "</span></a>"; }else{ - blob_appendf(p->pOut,"<s>"); - zTerm = "</s>"; + blob_appendf(p->pOut,"<span class=\"wikiTagCancelled\">"); + zTerm = "</span>"; } }else{ if( g.okHistory ){ blob_appendf(p->pOut,"<a href=\"%s/info/%s\">", g.zBaseURL, zTarget @@ -1039,11 +1064,11 @@ } } }else if( g.okHistory ){ blob_appendf(p->pOut, "<a href=\"%s/info/%s\">", g.zBaseURL, zTarget); } - }else if( strlen(zTarget)>=10 && isdigit(zTarget[0]) && zTarget[4]=='-' + }else if( strlen(zTarget)>=10 && fossil_isdigit(zTarget[0]) && zTarget[4]=='-' && db_int(0, "SELECT datetime(%Q) NOT NULL", zTarget) ){ blob_appendf(p->pOut, "<a href=\"%s/timeline?c=%T\">", g.zBaseURL, zTarget); }else if( strncmp(zTarget, "wiki:", 5)==0 && wiki_name_is_wellformed((const unsigned char*)zTarget) ){ zTarget += 5; @@ -1133,10 +1158,11 @@ }else{ if( p->wikiList!=MARKUP_UL ){ if( p->wikiList ){ popStackToTag(p, p->wikiList); } + endAutoParagraph(p); pushStack(p, MARKUP_UL); blob_append(p->pOut, "<ul>", 4); p->wikiList = MARKUP_UL; } popStackToTag(p, MARKUP_LI); @@ -1152,10 +1178,11 @@ }else{ if( p->wikiList!=MARKUP_OL ){ if( p->wikiList ){ popStackToTag(p, p->wikiList); } + endAutoParagraph(p); pushStack(p, MARKUP_OL); blob_append(p->pOut, "<ol>", 4); p->wikiList = MARKUP_OL; } popStackToTag(p, MARKUP_LI); @@ -1171,10 +1198,11 @@ }else{ if( p->wikiList!=MARKUP_OL ){ if( p->wikiList ){ popStackToTag(p, p->wikiList); } + endAutoParagraph(p); pushStack(p, MARKUP_OL); blob_append(p->pOut, "<ol>", 4); p->wikiList = MARKUP_OL; } popStackToTag(p, MARKUP_LI); @@ -1214,18 +1242,18 @@ zTarget = &z[1]; for(i=1; z[i] && z[i]!=']'; i++){ if( z[i]=='|' && zDisplay==0 ){ zDisplay = &z[i+1]; z[i] = 0; - for(j=i-1; j>0 && isspace(z[j]); j--){ z[j] = 0; } + for(j=i-1; j>0 && fossil_isspace(z[j]); j--){ z[j] = 0; } } } z[i] = 0; if( zDisplay==0 ){ zDisplay = zTarget; }else{ - while( isspace(*zDisplay) ) zDisplay++; + while( fossil_isspace(*zDisplay) ) zDisplay++; } openHyperlink(p, zTarget, zClose, sizeof(zClose)); savedState = p->state; p->state &= ~ALLOW_WIKI; p->state |= FONT_MARKUP_ONLY; @@ -1233,11 +1261,13 @@ p->state = savedState; blob_append(p->pOut, zClose, -1); break; } case TOKEN_TEXT: { - startAutoParagraph(p); + int i; + for(i=0; i<n && fossil_isspace(z[i]); i++){} + if( i<n ) startAutoParagraph(p); blob_append(p->pOut, z, n); break; } case TOKEN_RAW: { blob_append(p->pOut, z, n); @@ -1347,16 +1377,19 @@ blob_appendf(p->pOut, "<pre name='code' class='%s'>", markup.aAttr[vAttrIdx].zValue); vAttrDidAppend=1; } } - if( !vAttrDidAppend ) + if( !vAttrDidAppend ) { + endAutoParagraph(p); blob_append(p->pOut, "<pre class='verbatim'>",-1); + } p->wantAutoParagraph = 0; }else if( markup.iType==MUTYPE_LI ){ if( backupToType(p, MUTYPE_LIST)==0 ){ + endAutoParagraph(p); pushStack(p, MARKUP_UL); blob_append(p->pOut, "<ul>", 4); } pushStack(p, MARKUP_LI); renderMarkup(p->pOut, &markup); @@ -1384,12 +1417,22 @@ pushStack(p, markup.iCode); }else { if( markup.iType==MUTYPE_FONT ){ startAutoParagraph(p); - }else if( markup.iType==MUTYPE_BLOCK ){ + }else if( markup.iType==MUTYPE_BLOCK || markup.iType==MUTYPE_LIST ){ p->wantAutoParagraph = 0; + } + if( markup.iCode==MARKUP_HR + || markup.iCode==MARKUP_H1 + || markup.iCode==MARKUP_H2 + || markup.iCode==MARKUP_H3 + || markup.iCode==MARKUP_H4 + || markup.iCode==MARKUP_H5 + || markup.iCode==MARKUP_P + ){ + endAutoParagraph(p); } if( (markup.iType & MUTYPE_STACK )!=0 ){ pushStack(p, markup.iCode); } renderMarkup(p->pOut, &markup); @@ -1475,11 +1518,11 @@ int wiki_find_title(Blob *pIn, Blob *pTitle, Blob *pTail){ char *z; int i; int iStart; z = skip_bom(blob_str(pIn)); - for(i=0; isspace(z[i]); i++){} + for(i=0; fossil_isspace(z[i]); i++){} if( z[i]!='<' ) return 0; i++; if( strncmp(&z[i],"title>", 6)!=0 ) return 0; iStart = i+6; for(i=iStart; z[i] && (z[i]!='<' || strncmp(&z[i],"",8)!=0); i++){} Index: src/winhttp.c ================================================================== --- src/winhttp.c +++ src/winhttp.c @@ -16,12 +16,13 @@ ******************************************************************************* ** ** This file implements a very simple (and low-performance) HTTP server ** for windows. */ -#ifdef __MINGW32__ /* This code is for win32 only */ #include "config.h" +#ifdef _WIN32 +/* This code is for win32 only */ #include "winhttp.h" #include /* ** The HttpRequest structure holds information about each incoming @@ -105,14 +106,14 @@ wanted -= got; } fclose(out); out = 0; sprintf(zCmd, "\"%s\" http \"%s\" %s %s %s%s", - g.argv[0], g.zRepositoryName, zRequestFName, zReplyFName, + _pgmptr, g.zRepositoryName, zRequestFName, zReplyFName, inet_ntoa(p->addr.sin_addr), p->zNotFound ); - portable_system(zCmd); + fossil_system(zCmd); in = fopen(zReplyFName, "rb"); if( in ){ while( (got = fread(zHdr, 1, sizeof(zHdr), in))>0 ){ send(p->s, zHdr, got, 0); } @@ -133,11 +134,12 @@ */ void win32_http_server( int mnPort, int mxPort, /* Range of allowed TCP port numbers */ const char *zBrowser, /* Command to launch browser. (Or NULL) */ const char *zStopper, /* Stop server when this file is exists (Or NULL) */ - const char *zNotFound /* The --notfound option, or NULL */ + const char *zNotFound, /* The --notfound option, or NULL */ + int flags /* One or more HTTP_SERVER_ flags */ ){ WSADATA wd; SOCKET s = INVALID_SOCKET; SOCKADDR_IN addr; int idCnt = 0; @@ -158,11 +160,15 @@ if( s==INVALID_SOCKET ){ fossil_fatal("unable to create a socket"); } addr.sin_family = AF_INET; addr.sin_port = htons(iPort); - addr.sin_addr.s_addr = htonl(INADDR_ANY); + if( flags & HTTP_SERVER_LOCALHOST ){ + addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + }else{ + addr.sin_addr.s_addr = htonl(INADDR_ANY); + } if( bind(s, (struct sockaddr*)&addr, sizeof(addr))==SOCKET_ERROR ){ closesocket(s); iPort++; continue; } @@ -184,11 +190,11 @@ zTempPrefix = mprintf("fossil_server_P%d_", iPort); printf("Listening for HTTP requests on TCP port %d\n", iPort); if( zBrowser ){ zBrowser = mprintf(zBrowser, iPort); printf("Launch webbrowser: %s\n", zBrowser); - portable_system(zBrowser); + fossil_system(zBrowser); } printf("Type Ctrl-C to stop the HTTP server\n"); for(;;){ SOCKET client; SOCKADDR_IN client_addr; @@ -201,14 +207,11 @@ } if( client==INVALID_SOCKET ){ closesocket(s); fossil_fatal("error from accept()"); } - p = malloc( sizeof(*p) ); - if( p==0 ){ - fossil_fatal("out of memory"); - } + p = fossil_malloc( sizeof(*p) ); p->id = ++idCnt; p->s = client; p->addr = client_addr; p->zNotFound = zNotFoundOption; _beginthread(win32_process_one_http_request, 0, (void*)p); @@ -215,6 +218,6 @@ } closesocket(s); WSACleanup(); } -#endif /* __MINGW32__ -- This code is for win32 only */ +#endif /* _WIN32 -- This code is for win32 only */ Index: src/xfer.c ================================================================== --- src/xfer.c +++ src/xfer.c @@ -72,11 +72,17 @@ /* ** Remember that the other side of the connection already has a copy ** of the file rid. */ static void remote_has(int rid){ - if( rid ) db_multi_exec("INSERT OR IGNORE INTO onremote VALUES(%d)", rid); + if( rid ){ + static Stmt q; + db_static_prepare(&q, "INSERT OR IGNORE INTO onremote VALUES(:r)"); + db_bind_int(&q, ":r", rid); + db_step(&q); + db_reset(&q); + } } /* ** The aToken[0..nToken-1] blob array is a parse of a "file" line ** message. This routine finishes parsing that message and does @@ -95,11 +101,11 @@ ** be initialized to an empty string. ** ** Any artifact successfully received by this routine is considered to ** be public and is therefore removed from the "private" table. */ -static void xfer_accept_file(Xfer *pXfer){ +static void xfer_accept_file(Xfer *pXfer, int cloneFlag){ int n; int rid; int srcid = 0; Blob content, hash; @@ -114,27 +120,42 @@ return; } blob_zero(&content); blob_zero(&hash); blob_extract(pXfer->pIn, n, &content); - if( uuid_is_shunned(blob_str(&pXfer->aToken[1])) ){ + if( !cloneFlag && uuid_is_shunned(blob_str(&pXfer->aToken[1])) ){ /* Ignore files that have been shunned */ return; + } + if( cloneFlag ){ + if( pXfer->nToken==4 ){ + srcid = rid_from_uuid(&pXfer->aToken[2], 1); + pXfer->nDeltaRcvd++; + }else{ + srcid = 0; + pXfer->nFileRcvd++; + } + rid = content_put(&content, blob_str(&pXfer->aToken[1]), srcid); + remote_has(rid); + blob_reset(&content); + return; } if( pXfer->nToken==4 ){ - Blob src; + Blob src, next; srcid = rid_from_uuid(&pXfer->aToken[2], 1); if( content_get(srcid, &src)==0 ){ rid = content_put(&content, blob_str(&pXfer->aToken[1]), srcid); pXfer->nDanglingFile++; db_multi_exec("DELETE FROM phantom WHERE rid=%d", rid); content_make_public(rid); return; } pXfer->nDeltaRcvd++; - blob_delta_apply(&src, &content, &content); + blob_delta_apply(&src, &content, &next); blob_reset(&src); + blob_reset(&content); + content = next; }else{ pXfer->nFileRcvd++; } sha1sum_blob(&content, &hash); if( !blob_eq_str(&pXfer->aToken[1], blob_str(&hash), -1) ){ @@ -465,44 +486,56 @@ ** Check to see if the number of unclustered entries is greater than ** 100 and if it is, form a new cluster. Unclustered phantoms do not ** count toward the 100 total. And phantoms are never added to a new ** cluster. */ -static void create_cluster(void){ +void create_cluster(void){ Blob cluster, cksum; Stmt q; int nUncl; + int nRow = 0; /* We should not ever get any private artifacts in the unclustered table. ** But if we do (because of a bug) now is a good time to delete them. */ db_multi_exec( "DELETE FROM unclustered WHERE rid IN (SELECT rid FROM private)" ); - nUncl = db_int(0, "SELECT count(*) FROM unclustered" + nUncl = db_int(0, "SELECT count(*) FROM unclustered /*scan*/" " WHERE NOT EXISTS(SELECT 1 FROM phantom" " WHERE rid=unclustered.rid)"); - if( nUncl<100 ){ - return; - } - blob_zero(&cluster); - db_prepare(&q, "SELECT uuid FROM unclustered, blob" - " WHERE NOT EXISTS(SELECT 1 FROM phantom" - " WHERE rid=unclustered.rid)" - " AND unclustered.rid=blob.rid" - " AND NOT EXISTS(SELECT 1 FROM shun WHERE uuid=blob.uuid)" - " ORDER BY 1"); - while( db_step(&q)==SQLITE_ROW ){ - blob_appendf(&cluster, "M %s\n", db_column_text(&q, 0)); - } - db_finalize(&q); - md5sum_blob(&cluster, &cksum); - blob_appendf(&cluster, "Z %b\n", &cksum); - blob_reset(&cksum); - db_multi_exec("DELETE FROM unclustered"); - content_put(&cluster, 0, 0); - blob_reset(&cluster); + if( nUncl>=100 ){ + blob_zero(&cluster); + db_prepare(&q, "SELECT uuid FROM unclustered, blob" + " WHERE NOT EXISTS(SELECT 1 FROM phantom" + " WHERE rid=unclustered.rid)" + " AND unclustered.rid=blob.rid" + " AND NOT EXISTS(SELECT 1 FROM shun WHERE uuid=blob.uuid)" + " ORDER BY 1"); + while( db_step(&q)==SQLITE_ROW ){ + blob_appendf(&cluster, "M %s\n", db_column_text(&q, 0)); + nRow++; + if( nRow>=800 && nUncl>nRow+100 ){ + md5sum_blob(&cluster, &cksum); + blob_appendf(&cluster, "Z %b\n", &cksum); + blob_reset(&cksum); + content_put(&cluster, 0, 0); + blob_reset(&cluster); + nUncl -= nRow; + nRow = 0; + } + } + db_finalize(&q); + db_multi_exec("DELETE FROM unclustered"); + if( nRow>0 ){ + md5sum_blob(&cluster, &cksum); + blob_appendf(&cluster, "Z %b\n", &cksum); + blob_reset(&cksum); + content_put(&cluster, 0, 0); + blob_reset(&cluster); + } + } } /* ** Send an igot message for every entry in unclustered table. ** Return the number of cards sent. @@ -593,10 +626,11 @@ int deltaFlag = 0; int isClone = 0; int nGimme = 0; int size; int recvConfig = 0; + char *zNow; if( strcmp(PD("REQUEST_METHOD","POST"),"POST") ){ fossil_redirect_home(); } memset(&xfer, 0, sizeof(xfer)); @@ -603,11 +637,11 @@ blobarray_zero(xfer.aToken, count(xfer.aToken)); cgi_set_content_type(g.zContentType); blob_zero(&xfer.err); xfer.pIn = &g.cgiIn; xfer.pOut = cgi_output_blob(); - xfer.mxSend = db_get_int("max-download", 5000000); + xfer.mxSend = db_get_int("max-download", 20000000); g.xferPanic = 1; db_begin_transaction(); db_multi_exec( "CREATE TEMP TABLE onremote(rid INTEGER PRIMARY KEY);" @@ -627,11 +661,11 @@ cgi_reset_content(); @ error not\sauthorized\sto\swrite nErr++; break; } - xfer_accept_file(&xfer); + xfer_accept_file(&xfer, 0); if( blob_size(&xfer.err) ){ cgi_reset_content(); @ error %T(blob_str(&xfer.err)) nErr++; break; @@ -713,26 +747,42 @@ isPush = 1; } } }else - /* clone + /* clone ?PROTOCOL-VERSION? ?SEQUENCE-NUMBER? ** ** The client knows nothing. Tell all. */ if( blob_eq(&xfer.aToken[0], "clone") ){ + int iVers; login_check_credentials(); if( !g.okClone ){ cgi_reset_content(); @ push %s(db_get("server-code", "x")) %s(db_get("project-code", "x")) @ error not\sauthorized\sto\sclone nErr++; break; } - isClone = 1; - isPull = 1; - deltaFlag = 1; + if( xfer.nToken==3 + && blob_is_int(&xfer.aToken[1], &iVers) + && iVers>=2 + ){ + int seqno, max; + blob_is_int(&xfer.aToken[2], &seqno); + max = db_int(0, "SELECT max(rid) FROM blob"); + while( xfer.mxSend>blob_size(xfer.pOut) && seqno<=max ){ + send_file(&xfer, seqno, 0, 1); + seqno++; + } + if( seqno>=max ) seqno = 0; + @ clone_seqno %d(seqno) + }else{ + isClone = 1; + isPull = 1; + deltaFlag = 1; + } @ push %s(db_get("server-code", "x")) %s(db_get("project-code", "x")) }else /* login USER NONCE SIGNATURE ** @@ -869,10 +919,18 @@ } if( recvConfig ){ configure_finalize_receive(); } manifest_crosslink_end(); + + /* Send the server timestamp last, in case prior processing happened + ** to use up a significant fraction of our time window. + */ + zNow = db_text(0, "SELECT strftime('%%Y-%%m-%%dT%%H:%%M:%%S', 'now')"); + @ # timestamp %s(zNow) + free(zNow); + db_end_transaction(0); } /* ** COMMAND: test-xfer @@ -938,13 +996,17 @@ int origConfigRcvMask; /* Original value of configRcvMask */ int nFileRecv; /* Number of files received */ int mxPhantomReq = 200; /* Max number of phantoms to request per comm */ const char *zCookie; /* Server cookie */ int nSent, nRcvd; /* Bytes sent and received (after compression) */ + int cloneSeqno = 1; /* Sequence number for clones */ Blob send; /* Text we are sending to the server */ Blob recv; /* Reply we got back from the server */ Xfer xfer; /* Transfer data */ + int pctDone; /* Percentage done with a message */ + int lastPctDone = -1; /* Last displayed pctDone */ + double rArrivalTime; /* Time at which a message arrived */ const char *zSCode = db_get("server-code", "x"); const char *zPCode = db_get("project-code", 0); if( db_get_boolean("dont-push", 0) ) pushFlag = 0; if( pushFlag + pullFlag + cloneFlag == 0 @@ -972,24 +1034,26 @@ /* ** Always begin with a clone, pull, or push message */ if( cloneFlag ){ - blob_appendf(&send, "clone\n"); + blob_appendf(&send, "clone 2 %d\n", cloneSeqno); pushFlag = 0; pullFlag = 0; nCardSent++; /* TBD: Request all transferable configuration values */ + content_enable_dephantomize(0); }else if( pullFlag ){ blob_appendf(&send, "pull %s %s\n", zSCode, zPCode); nCardSent++; } if( pushFlag ){ blob_appendf(&send, "push %s %s\n", zSCode, zPCode); nCardSent++; } manifest_crosslink_begin(); + transport_global_startup(); fossil_print(zLabelFormat, "", "Bytes", "Cards", "Artifacts", "Deltas"); while( go ){ int newPhantom = 0; char *zRandomness; @@ -1003,11 +1067,11 @@ } /* Generate gimme cards for phantoms and leaf cards ** for all leaves. */ - if( pullFlag || cloneFlag ){ + if( pullFlag || (cloneFlag && cloneSeqno==1) ){ request_phantoms(&xfer, mxPhantomReq); } if( pushFlag ){ send_unsent(&xfer); nCardSent += send_unclustered(&xfer); @@ -1052,22 +1116,27 @@ blob_appendf(&send, "# %s\n", zRandomness); free(zRandomness); /* Exchange messages with the server */ nFileSend = xfer.nFileSent + xfer.nDeltaSent; - fossil_print(zValueFormat, "Send:", + fossil_print(zValueFormat, "Sent:", blob_size(&send), nCardSent+xfer.nGimmeSent+xfer.nIGotSent, xfer.nFileSent, xfer.nDeltaSent); nCardSent = 0; nCardRcvd = 0; xfer.nFileSent = 0; xfer.nDeltaSent = 0; xfer.nGimmeSent = 0; xfer.nIGotSent = 0; + if( !g.cgiOutput && !g.fQuiet ){ + printf("waiting for server..."); + } fflush(stdout); http_exchange(&send, &recv, cloneFlag==0 || nCycle>0); + lastPctDone = -1; blob_reset(&send); + rArrivalTime = db_double(0.0, "SELECT julianday('now')"); /* Begin constructing the next message (which might never be ** sent) by beginning with the pull or push cards */ if( pullFlag ){ @@ -1080,27 +1149,50 @@ } go = 0; /* Process the reply that came back from the server */ while( blob_line(&recv, &xfer.line) ){ + if( g.fHttpTrace ){ + printf("\rGOT: %.*s", (int)blob_size(&xfer.line), + blob_buffer(&xfer.line)); + } if( blob_buffer(&xfer.line)[0]=='#' ){ + const char *zLine = blob_buffer(&xfer.line); + if( memcmp(zLine, "# timestamp ", 12)==0 ){ + char zTime[20]; + double rDiff; + sqlite3_snprintf(sizeof(zTime), zTime, "%.19s", &zLine[12]); + rDiff = db_double(9e99, "SELECT julianday('%q') - %.17g", + zTime, rArrivalTime); + if( rDiff<0.0 ) rDiff = -rDiff; + if( rDiff>9e98 ) rDiff = 0.0; + if( (rDiff*24.0*3600.0)>=60.0 ){ + fossil_warning("*** time skew *** server time differs by %s", + db_timespan_name(rDiff)); + g.clockSkewSeen = 1; + } + } continue; } xfer.nToken = blob_tokenize(&xfer.line, xfer.aToken, count(xfer.aToken)); nCardRcvd++; - if( !g.cgiOutput && !g.fQuiet ){ - printf("\r%d", nCardRcvd); - fflush(stdout); + if( !g.cgiOutput && !g.fQuiet && recv.nUsed>0 ){ + pctDone = (recv.iCursor*100)/recv.nUsed; + if( pctDone!=lastPctDone ){ + printf("\rprocessed: %d%% ", pctDone); + lastPctDone = pctDone; + fflush(stdout); + } } /* file UUID SIZE \n CONTENT ** file UUID DELTASRC SIZE \n CONTENT ** ** Receive a file transmitted from the server. */ if( blob_eq(&xfer.aToken[0],"file") ){ - xfer_accept_file(&xfer); + xfer_accept_file(&xfer, cloneFlag); }else /* gimme UUID ** ** Server is requesting a file. If the file is a manifest, assume @@ -1156,11 +1248,11 @@ } if( zPCode==0 ){ zPCode = mprintf("%b", &xfer.aToken[2]); db_set("project-code", zPCode, 0); } - blob_appendf(&send, "clone\n"); + blob_appendf(&send, "clone 2 %d\n", cloneSeqno); nCardSent++; }else /* config NAME SIZE \n CONTENT ** @@ -1215,10 +1307,21 @@ ** same server. */ if( blob_eq(&xfer.aToken[0], "cookie") && xfer.nToken==2 ){ db_set("cookie", blob_str(&xfer.aToken[1]), 0); }else + + /* clone_seqno N + ** + ** When doing a clone, the server tries to send all of its artifacts + ** in sequence. This card indicates the sequence number of the next + ** blob that needs to be sent. If N<=0 that indicates that all blobs + ** have been sent. + */ + if( blob_eq(&xfer.aToken[0], "clone_seqno") && xfer.nToken==2 ){ + blob_is_int(&xfer.aToken[1], &cloneSeqno); + }else /* message MESSAGE ** ** Print a message. Similar to "error" but does not stop processing. ** @@ -1293,10 +1396,12 @@ nFileRecv = xfer.nFileRcvd + xfer.nDeltaRcvd + xfer.nDanglingFile; if( (nFileRecv>0 || newPhantom) && db_exists("SELECT 1 FROM phantom") ){ go = 1; mxPhantomReq = nFileRecv*2; if( mxPhantomReq<200 ) mxPhantomReq = 200; + }else if( cloneFlag && nFileRecv>0 ){ + go = 1; } nCardRcvd = 0; xfer.nFileRcvd = 0; xfer.nDeltaRcvd = 0; xfer.nDanglingFile = 0; @@ -1308,15 +1413,19 @@ go = 1; } /* If this is a clone, the go at least two rounds */ if( cloneFlag && nCycle==1 ) go = 1; + + /* Stop the cycle if the server sends a "clone_seqno 0" card */ + if( cloneSeqno<=0 ) go = 0; }; transport_stats(&nSent, &nRcvd, 1); fossil_print("Total network traffic: %d bytes sent, %d bytes received\n", nSent, nRcvd); transport_close(); transport_global_shutdown(); db_multi_exec("DROP TABLE onremote"); manifest_crosslink_end(); + content_enable_dephantomize(1); db_end_transaction(0); } Index: src/zip.c ================================================================== --- src/zip.c +++ src/zip.c @@ -102,11 +102,11 @@ for(j=0; j=nDir ){ nDir++; - azDir = realloc(azDir, sizeof(azDir[0])*nDir); + azDir = fossil_realloc(azDir, sizeof(azDir[0])*nDir); azDir[j] = mprintf("%s", zName); zip_add_file(zName, 0); } zName[i+1] = c; } @@ -313,64 +313,65 @@ ** politely expands into a subdir instead of filling your current dir ** with source files. For example, pass a UUID or "ProjectName". ** */ void zip_of_baseline(int rid, Blob *pZip, const char *zDir){ - int i; - Blob mfile, file, hash; - Manifest m; + Blob mfile, hash, file; + Manifest *pManifest; + ManifestFile *pFile; Blob filename; int nPrefix; content_get(rid, &mfile); if( blob_size(&mfile)==0 ){ blob_zero(pZip); return; } - blob_zero(&file); blob_zero(&hash); - blob_copy(&file, &mfile); blob_zero(&filename); zip_open(); if( zDir && zDir[0] ){ blob_appendf(&filename, "%s/", zDir); } nPrefix = blob_size(&filename); - if( manifest_parse(&m, &mfile) ){ + pManifest = manifest_get(rid, CFTYPE_MANIFEST); + if( pManifest ){ char *zName; - zip_set_timedate(m.rDate); - blob_append(&filename, "manifest", -1); - zName = blob_str(&filename); - zip_add_folders(zName); - zip_add_file(zName, &file); - sha1sum_blob(&file, &hash); - blob_reset(&file); - blob_append(&hash, "\n", 1); - blob_resize(&filename, nPrefix); - blob_append(&filename, "manifest.uuid", -1); - zName = blob_str(&filename); - zip_add_file(zName, &hash); - blob_reset(&hash); - for(i=0; irDate); + if( db_get_boolean("manifest", 0) ){ + blob_append(&filename, "manifest", -1); + zName = blob_str(&filename); + zip_add_folders(zName); + zip_add_file(zName, &mfile); + sha1sum_blob(&mfile, &hash); + blob_reset(&mfile); + blob_append(&hash, "\n", 1); + blob_resize(&filename, nPrefix); + blob_append(&filename, "manifest.uuid", -1); + zName = blob_str(&filename); + zip_add_file(zName, &hash); + blob_reset(&hash); + } + manifest_file_rewind(pManifest); + while( (pFile = manifest_file_next(pManifest,0))!=0 ){ + int fid = uuid_to_rid(pFile->zUuid, 0); if( fid ){ content_get(fid, &file); blob_resize(&filename, nPrefix); - blob_append(&filename, m.aFile[i].zName, -1); + blob_append(&filename, pFile->zName, -1); zName = blob_str(&filename); zip_add_folders(zName); zip_add_file(zName, &file); blob_reset(&file); } } - manifest_clear(&m); }else{ blob_reset(&mfile); - blob_reset(&file); } + manifest_destroy(pManifest); blob_reset(&filename); zip_close(pZip); } /* Index: test/delta1.test ================================================================== --- test/delta1.test +++ test/delta1.test @@ -32,11 +32,11 @@ # work properly. # set filelist [glob $testdir/*] foreach f $filelist { set base [file root [file tail $f]] -puts "base=$base f=$f" +protOut "base=$base f=$f" set f1 [read_file $f] write_file t1 $f1 for {set i 0} {$i<100} {incr i} { write_file t2 [random_changes $f1 1 1 0 0.1] fossil test-delta t1 t2 Index: test/merge1.test ================================================================== --- test/merge1.test +++ test/merge1.test @@ -77,26 +77,26 @@ 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } write_file_indented t23 { - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT 111 - This is line ONE of the demo program - 1111 ============================ 111 - This is line one OF the demo program - 1111 - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT 222 - The second line program line in code - 2222 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } write_file_indented t32 { - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT 111 - This is line one OF the demo program - 1111 ============================ 111 - This is line ONE of the demo program - 1111 - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT 222 - The second line program line in code - 2222 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } @@ -158,26 +158,26 @@ 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } write_file_indented t32 { - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT ============================ 000 - Zero lines added to the beginning of - 0000 111 - This is line one of the demo program - 1111 - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT 222 - The second line program line in code - 2222 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } write_file_indented t23 { - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT 000 - Zero lines added to the beginning of - 0000 111 - This is line one of the demo program - 1111 ============================ - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT 222 - The second line program line in code - 2222 333 - This is a test of the merging algohm - 3333 444 - If all goes well, we will be pleased - 4444 555 - we think it well and other stuff too - 5555 } @@ -293,11 +293,11 @@ STUV XYZ. } write_file_indented t23 { abcd - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT efgh 2 ijkl 2 mnop 2 qrst uvwx @@ -311,11 +311,11 @@ qrst 3 uvwx 3 yzAB 3 CDEF GHIJ - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT KLMN OPQR STUV XYZ. } @@ -354,11 +354,11 @@ } write_file_indented t23 { abcd efgh 2 ijkl 2 - >>>>>>> BEGIN MERGE CONFLICT + <<<<<<< BEGIN MERGE CONFLICT mnop qrst uvwx yzAB 2 CDEF 2 @@ -368,13 +368,13 @@ qrst 3 uvwx 3 yzAB 3 CDEF GHIJ - <<<<<<< END MERGE CONFLICT + >>>>>>> END MERGE CONFLICT KLMN OPQR STUV XYZ. } fossil test-3 t1 t2 t3 a23 test merge1-7.2 {[same_file t23 a23]} Index: test/merge3.test ================================================================== --- test/merge3.test +++ test/merge3.test @@ -28,18 +28,18 @@ write_file t1 [join [string trim $basis] \n]\n write_file t2 [join [string trim $v1] \n]\n write_file t3 [join [string trim $v2] \n]\n fossil test-3-way-merge t1 t2 t3 t4 set x [read_file t4] - regsub -all {>>>>>>> BEGIN MERGE CONFLICT} $x {>} x + regsub -all {<<<<<<< BEGIN MERGE CONFLICT} $x {>} x regsub -all {============================} $x {=} x - regsub -all {<<<<<<< END MERGE CONFLICT} $x {<} x + regsub -all {>>>>>>> END MERGE CONFLICT} $x {<} x set x [split [string trim $x] \n] set result [string trim $result] if {$x!=$result} { - puts " Expected \[$result\]" - puts " Got \[$x\]" + protOut " Expected \[$result\]" + protOut " Got \[$x\]" test merge3-$testid 0 } else { test merge3-$testid 1 } } Index: test/merge4.test ================================================================== --- test/merge4.test +++ test/merge4.test @@ -29,29 +29,29 @@ write_file t2 [join [string trim $v1] \n]\n write_file t3 [join [string trim $v2] \n]\n fossil test-3-way-merge t1 t2 t3 t4 fossil test-3-way-merge t1 t3 t2 t5 set x [read_file t4] - regsub -all {>>>>>>> BEGIN MERGE CONFLICT} $x {>} x + regsub -all {<<<<<<< BEGIN MERGE CONFLICT} $x {>} x regsub -all {============================} $x {=} x - regsub -all {<<<<<<< END MERGE CONFLICT} $x {<} x + regsub -all {>>>>>>> END MERGE CONFLICT} $x {<} x set x [split [string trim $x] \n] set y [read_file t5] - regsub -all {>>>>>>> BEGIN MERGE CONFLICT} $y {>} y + regsub -all {<<<<<<< BEGIN MERGE CONFLICT} $y {>} y regsub -all {============================} $y {=} y - regsub -all {<<<<<<< END MERGE CONFLICT} $y {<} y + regsub -all {>>>>>>> END MERGE CONFLICT} $y {<} y set y [split [string trim $y] \n] set result1 [string trim $result1] if {$x!=$result1} { - puts " Expected \[$result1\]" - puts " Got \[$x\]" + protOut " Expected \[$result1\]" + protOut " Got \[$x\]" test merge3-$testid 0 } else { set result2 [string trim $result2] if {$y!=$result2} { - puts " Expected \[$result2\]" - puts " Got \[$y\]" + protOut " Expected \[$result2\]" + protOut " Got \[$y\]" test merge3-$testid 0 } else { test merge3-$testid 1 } } Index: test/tester.tcl ================================================================== --- test/tester.tcl +++ test/tester.tcl @@ -38,27 +38,59 @@ set HALT 1 set argv [lreplace $argv $i $i] } else { set HALT 0 } + +set i [lsearch $argv -prot] +if {$i>=0} { + set PROT 1 + set argv [lreplace $argv $i $i] +} else { + set PROT 0 +} if {[llength $argv]==0} { foreach f [lsort [glob $testdir/*.test]] { set base [file root [file tail $f]] lappend argv $base } } + +# start protocol +# +proc protInit {cmd} { + if {$::PROT} { + set out [open "prot" w] + fconfigure $out -translation platform + puts $out "starting tests with:$cmd" + close $out + } +} + +# write protocol +# +proc protOut {msg} { + puts "$msg" + if {$::PROT} { + set out [open "prot" a] + fconfigure $out -translation platform + puts $out "$msg" + close $out + } +} # Run the fossil program # proc fossil {args} { global fossilexe set cmd $fossilexe foreach a $args { lappend cmd $a } - puts $cmd + protOut $cmd + flush stdout set rc [catch {eval exec $cmd} result] global RESULT CODE set CODE $rc set RESULT $result @@ -100,13 +132,13 @@ # proc test {name expr} { global bad_test set r [uplevel 1 [list expr $expr]] if {$r} { - puts "test $name OK" + protOut "test $name OK" } else { - puts "test $name FAILED!" + protOut "test $name FAILED!" lappend bad_test $name if {$::HALT} exit } } set bad_test {} @@ -167,10 +199,11 @@ append out \n$line } return [string range $out 1 end] } +protInit $fossilexe foreach testfile $argv { - puts "***** $testfile ******" + protOut "***** $testfile ******" source $testdir/$testfile.test } -puts "[llength $bad_test] errors: $bad_test" +protOut "[llength $bad_test] errors: $bad_test" ADDED win/Makefile.PellesCGMake Index: win/Makefile.PellesCGMake ================================================================== --- win/Makefile.PellesCGMake +++ win/Makefile.PellesCGMake @@ -0,0 +1,186 @@ +# ########################################################################### +# +# HowTo +# ----- +# +# This is a Makefile to compile fossil with PellesC from +# http://www.smorgasbordet.com/pellesc/index.htm +# In addition to the Compiler envrionment, you need +# gmake from http://sourceforge.net/projects/unxutils/, Pelles make version +# couldn't handle the complex dependencies in this build +# zlib sources +# Then you do +# 1. create a directory PellesC in the project root directory +# 2. Change the variables PellesCDir/ZLIBSRCDIR to the path of your installation +# 3. open a dos prompt window and change working directory into PellesC (step 1) +# 4. run gmake -f ..\win\Makefile.PellesCGMake +# +# this file is tested with +# PellesC 5.00.13 +# gmake 3.80 +# zlib sources 1.2.5 +# Windows XP SP 2 +# and +# PellesC 6.00.4 +# gmake 3.80 +# zlib sources 1.2.5 +# Windows 7 Home Premium +# +# ########################################################################### + +# +PellesCDir=c:\Programme\PellesC + +# Select between 32/64 bit code, default is 32 bit +#TARGETVERSION=64 + +ifeq ($(TARGETVERSION),64) +# 64 bit version +TARGETMACHINE_CC=amd64 +TARGETMACHINE_LN=amd64 +TARGETEXTEND=64 +else +# 32 bit version +TARGETMACHINE_CC=x86 +TARGETMACHINE_LN=ix86 +TARGETEXTEND= +endif + +# define the project directories +B=.. +SRCDIR=$(B)/src/ +WINDIR=$(B)/win/ +ZLIBSRCDIR=../../zlib/ + +# define linker command and options +LINK=$(PellesCDir)/bin/polink.exe +LINKFLAGS=-subsystem:console -machine:$(TARGETMACHINE_LN) /LIBPATH:$(PellesCDir)\lib\win$(TARGETEXTEND) /LIBPATH:$(PellesCDir)\lib kernel32.lib advapi32.lib delayimp$(TARGETEXTEND).lib Wsock32.lib Crtmt$(TARGETEXTEND).lib + +# define standard C-compiler and flags, used to compile +# the fossil binary. Some special definitions follow for +# special files follow +CC=$(PellesCDir)\bin\pocc.exe +DEFINES=-DFOSSIL_I18N=0 -Dstrncasecmp=memicmp -Dstrcasecmp=stricmp +CCFLAGS=-T$(TARGETMACHINE_CC)-coff -Ot -W2 -Gd -Go -Ze -MT $(DEFINES) +INCLUDE=/I $(PellesCDir)\Include\Win /I $(PellesCDir)\Include /I $(ZLIBSRCDIR) /I $(SRCDIR) + +# define commands for building the windows resource files +RESOURCE=fossil.res +RC=$(PellesCDir)\bin\porc.exe +RCFLAGS=$(INCLUDE) -D__POCC__=1 -D_M_X$(TARGETVERSION) + +# define the special utilities files, needed to generate +# the automatically generated source files +UTILS=translate.exe mkindex.exe makeheaders.exe +UTILS_OBJ=$(UTILS:.exe=.obj) +UTILS_SRC=$(foreach uf,$(UTILS),$(SRCDIR)$(uf:.exe=.c)) + +# define the sqlite files, which need special flags on compile +SQLITESRC=sqlite3.c +ORIGSQLITESRC=$(foreach sf,$(SQLITESRC),$(SRCDIR)$(sf)) +SQLITEOBJ=$(foreach sf,$(SQLITESRC),$(sf:.c=.obj)) +SQLITEDEFINES=-DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 + +# define the th scripting files, which need special flags on compile +THSRC=th.c th_lang.c +ORIGTHSRC=$(foreach sf,$(THSRC),$(SRCDIR)$(sf)) +THOBJ=$(foreach sf,$(THSRC),$(sf:.c=.obj)) + +# define the zlib files, needed by this compile +ZLIBSRC=adler32.c compress.c crc32.c deflate.c gzclose.c gzlib.c gzread.c gzwrite.c infback.c inffast.c inflate.c inftrees.c trees.c uncompr.c zutil.c +ORIGZLIBSRC=$(foreach sf,$(ZLIBSRC),$(ZLIBSRCDIR)$(sf)) +ZLIBOBJ=$(foreach sf,$(ZLIBSRC),$(sf:.c=.obj)) + +# define all fossil sources, using the standard compile and +# source generation. These are all files in SRCDIR, which are not +# mentioned as special files above: +ORIGSRC=$(filter-out $(UTILS_SRC) $(ORIGTHSRC) $(ORIGSQLITESRC),$(wildcard $(SRCDIR)*.c)) +SRC=$(subst $(SRCDIR),,$(ORIGSRC)) +TRANSLATEDSRC=$(SRC:.c=_.c) +TRANSLATEDOBJ=$(TRANSLATEDSRC:.c=.obj) + +# main target file is the application +APPLICATION=fossil.exe + +# ########################################################################### +# define the standard make target +.PHONY: default +default: page_index.h headers $(APPLICATION) + +# ########################################################################### +# symbolic target to generate the source generate utils +.PHONY: utils +utils: $(UTILS) + +# link utils +$(UTILS) version.exe: %.exe: %.obj + $(LINK) $(LINKFLAGS) -out:"$@" $< + +# compiling standard fossil utils +$(UTILS_OBJ): %.obj: $(SRCDIR)%.c + $(CC) $(CCFLAGS) $(INCLUDE) "$<" -Fo"$@" + +# compile special windows utils +version.obj: $(WINDIR)version.c + $(CC) $(CCFLAGS) $(INCLUDE) "$<" -Fo"$@" + +# ########################################################################### +# generate the translated c-source files +$(TRANSLATEDSRC): %_.c: $(SRCDIR)%.c translate.exe + translate.exe $< >$@ + +# ########################################################################### +# generate the index source, containing all web references,.. +page_index.h: $(TRANSLATEDSRC) mkindex.exe + mkindex.exe $(TRANSLATEDSRC) >$@ + +# ########################################################################### +# extracting version info from manifest +VERSION.h: version.exe ..\manifest.uuid ..\manifest + version.exe ..\manifest.uuid ..\manifest > $@ + +# ########################################################################### +# generate the simplified headers +headers: makeheaders.exe page_index.h VERSION.h ../src/sqlite3.h ../src/th.h VERSION.h + makeheaders.exe $(foreach ts,$(TRANSLATEDSRC),$(ts):$(ts:_.c=.h)) ../src/sqlite3.h ../src/th.h VERSION.h + echo Done >$@ + +# ########################################################################### +# compile C sources with relevant options + +$(TRANSLATEDOBJ): %_.obj: %_.c %.h + $(CC) $(CCFLAGS) $(INCLUDE) "$<" -Fo"$@" + +$(SQLITEOBJ): %.obj: $(SRCDIR)%.c $(SRCDIR)%.h + $(CC) $(CCFLAGS) $(SQLITEDEFINES) $(INCLUDE) "$<" -Fo"$@" + +$(THOBJ): %.obj: $(SRCDIR)%.c $(SRCDIR)th.h + $(CC) $(CCFLAGS) $(INCLUDE) "$<" -Fo"$@" + +$(ZLIBOBJ): %.obj: $(ZLIBSRCDIR)%.c + $(CC) $(CCFLAGS) $(INCLUDE) "$<" -Fo"$@" + +# ########################################################################### +# create the windows resource with icon and version info +$(RESOURCE): %.res: ../win/%.rc ../win/*.ico + $(RC) $(RCFLAGS) $< -Fo"$@" + +# ########################################################################### +# link the application +$(APPLICATION): $(TRANSLATEDOBJ) $(SQLITEOBJ) $(THOBJ) $(ZLIBOBJ) headers $(RESOURCE) + $(LINK) $(LINKFLAGS) -out:"$@" $(TRANSLATEDOBJ) $(SQLITEOBJ) $(THOBJ) $(ZLIBOBJ) $(RESOURCE) + +# ########################################################################### +# cleanup + +.PHONY: clean +clean: + del /F $(TRANSLATEDOBJ) $(SQLITEOBJ) $(THOBJ) $(ZLIBOBJ) $(UTILS_OBJ) version.obj + del /F $(TRANSLATEDSRC) + del /F *.h headers + del /F $(RESOURCE) + +.PHONY: clobber +clobber: clean + del /F *.exe + ADDED win/Makefile.dmc Index: win/Makefile.dmc ================================================================== --- win/Makefile.dmc +++ win/Makefile.dmc @@ -0,0 +1,535 @@ +# DO NOT EDIT +# +# This file is automatically generated. Instead of editing this +# file, edit "makemake.tcl" then run +# "tclsh src/makemake.tcl dmc > win/Makefile.dmc" +# to regenerate this file. +B = .. +SRCDIR = $B\src +OBJDIR = . +O = .obj +E = .exe + + +# Maybe DMDIR, SSL or INCL needs adjustment +DMDIR = c:\DM +INCL = -I. -I$(SRCDIR) -I$B\win\include -I$(DMDIR)\extra\include + +#SSL = -DFOSSIL_ENABLE_SSL=1 +SSL = + +DMCDEF = -Dstrncasecmp=memicmp -Dstrcasecmp=stricmp +I18N = -DFOSSIL_I18N=0 + +CFLAGS = -o +BCC = $(DMDIR)\bin\dmc $(CFLAGS) +TCC = $(DMDIR)\bin\dmc $(CFLAGS) $(DMCDEF) $(I18N) $(SSL) $(INCL) +LIBS = $(DMDIR)\extra\lib\ zlib wsock32 + +SRC = add_.c allrepo_.c attach_.c bag_.c blob_.c branch_.c browse_.c captcha_.c cgi_.c checkin_.c checkout_.c clearsign_.c clone_.c comformat_.c configure_.c content_.c db_.c delta_.c deltacmd_.c descendants_.c diff_.c diffcmd_.c doc_.c encode_.c event_.c file_.c finfo_.c graph_.c http_.c http_socket_.c http_ssl_.c http_transport_.c info_.c login_.c main_.c manifest_.c md5_.c merge_.c merge3_.c name_.c pivot_.c popen_.c pqueue_.c printf_.c rebuild_.c report_.c rss_.c schema_.c search_.c setup_.c sha1_.c shun_.c skins_.c stat_.c style_.c sync_.c tag_.c th_main_.c timeline_.c tkt_.c tktsetup_.c undo_.c update_.c url_.c user_.c verify_.c vfile_.c wiki_.c wikiformat_.c winhttp_.c xfer_.c zip_.c + +OBJ = $(OBJDIR)\add$O $(OBJDIR)\allrepo$O $(OBJDIR)\attach$O $(OBJDIR)\bag$O $(OBJDIR)\blob$O $(OBJDIR)\branch$O $(OBJDIR)\browse$O $(OBJDIR)\captcha$O $(OBJDIR)\cgi$O $(OBJDIR)\checkin$O $(OBJDIR)\checkout$O $(OBJDIR)\clearsign$O $(OBJDIR)\clone$O $(OBJDIR)\comformat$O $(OBJDIR)\configure$O $(OBJDIR)\content$O $(OBJDIR)\db$O $(OBJDIR)\delta$O $(OBJDIR)\deltacmd$O $(OBJDIR)\descendants$O $(OBJDIR)\diff$O $(OBJDIR)\diffcmd$O $(OBJDIR)\doc$O $(OBJDIR)\encode$O $(OBJDIR)\event$O $(OBJDIR)\file$O $(OBJDIR)\finfo$O $(OBJDIR)\graph$O $(OBJDIR)\http$O $(OBJDIR)\http_socket$O $(OBJDIR)\http_ssl$O $(OBJDIR)\http_transport$O $(OBJDIR)\info$O $(OBJDIR)\login$O $(OBJDIR)\main$O $(OBJDIR)\manifest$O $(OBJDIR)\md5$O $(OBJDIR)\merge$O $(OBJDIR)\merge3$O $(OBJDIR)\name$O $(OBJDIR)\pivot$O $(OBJDIR)\popen$O $(OBJDIR)\pqueue$O $(OBJDIR)\printf$O $(OBJDIR)\rebuild$O $(OBJDIR)\report$O $(OBJDIR)\rss$O $(OBJDIR)\schema$O $(OBJDIR)\search$O $(OBJDIR)\setup$O $(OBJDIR)\sha1$O $(OBJDIR)\shun$O $(OBJDIR)\skins$O $(OBJDIR)\stat$O $(OBJDIR)\style$O $(OBJDIR)\sync$O $(OBJDIR)\tag$O $(OBJDIR)\th_main$O $(OBJDIR)\timeline$O $(OBJDIR)\tkt$O $(OBJDIR)\tktsetup$O $(OBJDIR)\undo$O $(OBJDIR)\update$O $(OBJDIR)\url$O $(OBJDIR)\user$O $(OBJDIR)\verify$O $(OBJDIR)\vfile$O $(OBJDIR)\wiki$O $(OBJDIR)\wikiformat$O $(OBJDIR)\winhttp$O $(OBJDIR)\xfer$O $(OBJDIR)\zip$O $(OBJDIR)\sqlite3$O $(OBJDIR)\shell$O $(OBJDIR)\th$O $(OBJDIR)\th_lang$O + +RC=$(DMDIR)\bin\rcc +RCFLAGS=-32 -w1 -I$(SRCDIR) /D__DMC__ + +APPNAME = $(OBJDIR)\fossil$(E) + +all: $(APPNAME) + +$(APPNAME) : translate$E mkindex$E headers fossil.res $(OBJ) $(OBJDIR)\link + cd $(OBJDIR) + $(DMDIR)\bin\link @link + +fossil.res: $B\win\fossil.rc + $(RC) $(RCFLAGS) -o$@ $** + +$(OBJDIR)\link: $B\win\Makefile.dmc + +echo add allrepo attach bag blob branch browse captcha cgi checkin checkout clearsign clone comformat configure content db delta deltacmd descendants diff diffcmd doc encode event file finfo graph http http_socket http_ssl http_transport info login main manifest md5 merge merge3 name pivot popen pqueue printf rebuild report rss schema search setup sha1 shun skins stat style sync tag th_main timeline tkt tktsetup undo update url user verify vfile wiki wikiformat winhttp xfer zip sqlite3 shell th th_lang > $@ + +echo fossil >> $@ + +echo fossil >> $@ + +echo $(LIBS) >> $@ + +echo. >> $@ + +echo fossil >> $@ + + + +translate$E: $(SRCDIR)\translate.c + $(BCC) -o$@ $** + +makeheaders$E: $(SRCDIR)\makeheaders.c + $(BCC) -o$@ $** + +mkindex$E: $(SRCDIR)\mkindex.c + $(BCC) -o$@ $** + +version$E: $B\win\version.c + $(BCC) -o$@ $** + +$(OBJDIR)\shell$O : $(SRCDIR)\shell.c + $(TCC) -o$@ -c -Dmain=sqlite3_shell -DSQLITE_OMIT_LOAD_EXTENSION=1 $** + +$(OBJDIR)\sqlite3$O : $(SRCDIR)\sqlite3.c + $(TCC) -o$@ -c -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 $** + +$(OBJDIR)\th$O : $(SRCDIR)\th.c + $(TCC) -o$@ -c $** + +$(OBJDIR)\th_lang$O : $(SRCDIR)\th_lang.c + $(TCC) -o$@ -c $** + +VERSION.h : version$E $B\manifest.uuid $B\manifest + +$** > $@ + +page_index.h: mkindex$E $(SRC) + +$** > $@ + +clean: + -del $(OBJDIR)\*.obj + -del *.obj *_.c *.h *.map + +realclean: + -del $(APPNAME) translate$E mkindex$E makeheaders$E version$E + + +$(OBJDIR)\add$O : add_.c add.h + $(TCC) -o$@ -c add_.c + +add_.c : $(SRCDIR)\add.c + +translate$E $** > $@ + +$(OBJDIR)\allrepo$O : allrepo_.c allrepo.h + $(TCC) -o$@ -c allrepo_.c + +allrepo_.c : $(SRCDIR)\allrepo.c + +translate$E $** > $@ + +$(OBJDIR)\attach$O : attach_.c attach.h + $(TCC) -o$@ -c attach_.c + +attach_.c : $(SRCDIR)\attach.c + +translate$E $** > $@ + +$(OBJDIR)\bag$O : bag_.c bag.h + $(TCC) -o$@ -c bag_.c + +bag_.c : $(SRCDIR)\bag.c + +translate$E $** > $@ + +$(OBJDIR)\blob$O : blob_.c blob.h + $(TCC) -o$@ -c blob_.c + +blob_.c : $(SRCDIR)\blob.c + +translate$E $** > $@ + +$(OBJDIR)\branch$O : branch_.c branch.h + $(TCC) -o$@ -c branch_.c + +branch_.c : $(SRCDIR)\branch.c + +translate$E $** > $@ + +$(OBJDIR)\browse$O : browse_.c browse.h + $(TCC) -o$@ -c browse_.c + +browse_.c : $(SRCDIR)\browse.c + +translate$E $** > $@ + +$(OBJDIR)\captcha$O : captcha_.c captcha.h + $(TCC) -o$@ -c captcha_.c + +captcha_.c : $(SRCDIR)\captcha.c + +translate$E $** > $@ + +$(OBJDIR)\cgi$O : cgi_.c cgi.h + $(TCC) -o$@ -c cgi_.c + +cgi_.c : $(SRCDIR)\cgi.c + +translate$E $** > $@ + +$(OBJDIR)\checkin$O : checkin_.c checkin.h + $(TCC) -o$@ -c checkin_.c + +checkin_.c : $(SRCDIR)\checkin.c + +translate$E $** > $@ + +$(OBJDIR)\checkout$O : checkout_.c checkout.h + $(TCC) -o$@ -c checkout_.c + +checkout_.c : $(SRCDIR)\checkout.c + +translate$E $** > $@ + +$(OBJDIR)\clearsign$O : clearsign_.c clearsign.h + $(TCC) -o$@ -c clearsign_.c + +clearsign_.c : $(SRCDIR)\clearsign.c + +translate$E $** > $@ + +$(OBJDIR)\clone$O : clone_.c clone.h + $(TCC) -o$@ -c clone_.c + +clone_.c : $(SRCDIR)\clone.c + +translate$E $** > $@ + +$(OBJDIR)\comformat$O : comformat_.c comformat.h + $(TCC) -o$@ -c comformat_.c + +comformat_.c : $(SRCDIR)\comformat.c + +translate$E $** > $@ + +$(OBJDIR)\configure$O : configure_.c configure.h + $(TCC) -o$@ -c configure_.c + +configure_.c : $(SRCDIR)\configure.c + +translate$E $** > $@ + +$(OBJDIR)\content$O : content_.c content.h + $(TCC) -o$@ -c content_.c + +content_.c : $(SRCDIR)\content.c + +translate$E $** > $@ + +$(OBJDIR)\db$O : db_.c db.h + $(TCC) -o$@ -c db_.c + +db_.c : $(SRCDIR)\db.c + +translate$E $** > $@ + +$(OBJDIR)\delta$O : delta_.c delta.h + $(TCC) -o$@ -c delta_.c + +delta_.c : $(SRCDIR)\delta.c + +translate$E $** > $@ + +$(OBJDIR)\deltacmd$O : deltacmd_.c deltacmd.h + $(TCC) -o$@ -c deltacmd_.c + +deltacmd_.c : $(SRCDIR)\deltacmd.c + +translate$E $** > $@ + +$(OBJDIR)\descendants$O : descendants_.c descendants.h + $(TCC) -o$@ -c descendants_.c + +descendants_.c : $(SRCDIR)\descendants.c + +translate$E $** > $@ + +$(OBJDIR)\diff$O : diff_.c diff.h + $(TCC) -o$@ -c diff_.c + +diff_.c : $(SRCDIR)\diff.c + +translate$E $** > $@ + +$(OBJDIR)\diffcmd$O : diffcmd_.c diffcmd.h + $(TCC) -o$@ -c diffcmd_.c + +diffcmd_.c : $(SRCDIR)\diffcmd.c + +translate$E $** > $@ + +$(OBJDIR)\doc$O : doc_.c doc.h + $(TCC) -o$@ -c doc_.c + +doc_.c : $(SRCDIR)\doc.c + +translate$E $** > $@ + +$(OBJDIR)\event$O : event_.c event.h + $(TCC) -o$@ -c event_.c + +event_.c : $(SRCDIR)\event.c + +translate$E $** > $@ + +$(OBJDIR)\encode$O : encode_.c encode.h + $(TCC) -o$@ -c encode_.c + +encode_.c : $(SRCDIR)\encode.c + +translate$E $** > $@ + +$(OBJDIR)\event$O : event_.c event.h + $(TCC) -o$@ -c event_.c + +event_.c : $(SRCDIR)\event.c + +translate$E $** > $@ + +$(OBJDIR)\file$O : file_.c file.h + $(TCC) -o$@ -c file_.c + +file_.c : $(SRCDIR)\file.c + +translate$E $** > $@ + +$(OBJDIR)\finfo$O : finfo_.c finfo.h + $(TCC) -o$@ -c finfo_.c + +finfo_.c : $(SRCDIR)\finfo.c + +translate$E $** > $@ + +$(OBJDIR)\graph$O : graph_.c graph.h + $(TCC) -o$@ -c graph_.c + +graph_.c : $(SRCDIR)\graph.c + +translate$E $** > $@ + +$(OBJDIR)\http$O : http_.c http.h + $(TCC) -o$@ -c http_.c + +http_.c : $(SRCDIR)\http.c + +translate$E $** > $@ + +$(OBJDIR)\http_socket$O : http_socket_.c http_socket.h + $(TCC) -o$@ -c http_socket_.c + +http_socket_.c : $(SRCDIR)\http_socket.c + +translate$E $** > $@ + +$(OBJDIR)\http_ssl$O : http_ssl_.c http_ssl.h + $(TCC) -o$@ -c http_ssl_.c + +http_ssl_.c : $(SRCDIR)\http_ssl.c + +translate$E $** > $@ + +$(OBJDIR)\http_transport$O : http_transport_.c http_transport.h + $(TCC) -o$@ -c http_transport_.c + +http_transport_.c : $(SRCDIR)\http_transport.c + +translate$E $** > $@ + +$(OBJDIR)\info$O : info_.c info.h + $(TCC) -o$@ -c info_.c + +info_.c : $(SRCDIR)\info.c + +translate$E $** > $@ + +$(OBJDIR)\login$O : login_.c login.h + $(TCC) -o$@ -c login_.c + +login_.c : $(SRCDIR)\login.c + +translate$E $** > $@ + +$(OBJDIR)\main$O : main_.c main.h + $(TCC) -o$@ -c main_.c + +main_.c : $(SRCDIR)\main.c + +translate$E $** > $@ + +$(OBJDIR)\manifest$O : manifest_.c manifest.h + $(TCC) -o$@ -c manifest_.c + +manifest_.c : $(SRCDIR)\manifest.c + +translate$E $** > $@ + +$(OBJDIR)\md5$O : md5_.c md5.h + $(TCC) -o$@ -c md5_.c + +md5_.c : $(SRCDIR)\md5.c + +translate$E $** > $@ + +$(OBJDIR)\merge$O : merge_.c merge.h + $(TCC) -o$@ -c merge_.c + +merge_.c : $(SRCDIR)\merge.c + +translate$E $** > $@ + +$(OBJDIR)\merge3$O : merge3_.c merge3.h + $(TCC) -o$@ -c merge3_.c + +merge3_.c : $(SRCDIR)\merge3.c + +translate$E $** > $@ + +$(OBJDIR)\name$O : name_.c name.h + $(TCC) -o$@ -c name_.c + +name_.c : $(SRCDIR)\name.c + +translate$E $** > $@ + +$(OBJDIR)\pivot$O : pivot_.c pivot.h + $(TCC) -o$@ -c pivot_.c + +pivot_.c : $(SRCDIR)\pivot.c + +translate$E $** > $@ + +$(OBJDIR)\popen$O : popen_.c popen.h + $(TCC) -o$@ -c popen_.c + +popen_.c : $(SRCDIR)\popen.c + +translate$E $** > $@ + +$(OBJDIR)\pqueue$O : pqueue_.c pqueue.h + $(TCC) -o$@ -c pqueue_.c + +pqueue_.c : $(SRCDIR)\pqueue.c + +translate$E $** > $@ + +$(OBJDIR)\printf$O : printf_.c printf.h + $(TCC) -o$@ -c printf_.c + +printf_.c : $(SRCDIR)\printf.c + +translate$E $** > $@ + +$(OBJDIR)\rebuild$O : rebuild_.c rebuild.h + $(TCC) -o$@ -c rebuild_.c + +rebuild_.c : $(SRCDIR)\rebuild.c + +translate$E $** > $@ + +$(OBJDIR)\report$O : report_.c report.h + $(TCC) -o$@ -c report_.c + +report_.c : $(SRCDIR)\report.c + +translate$E $** > $@ + +$(OBJDIR)\rss$O : rss_.c rss.h + $(TCC) -o$@ -c rss_.c + +rss_.c : $(SRCDIR)\rss.c + +translate$E $** > $@ + +$(OBJDIR)\schema$O : schema_.c schema.h + $(TCC) -o$@ -c schema_.c + +schema_.c : $(SRCDIR)\schema.c + +translate$E $** > $@ + +$(OBJDIR)\search$O : search_.c search.h + $(TCC) -o$@ -c search_.c + +search_.c : $(SRCDIR)\search.c + +translate$E $** > $@ + +$(OBJDIR)\setup$O : setup_.c setup.h + $(TCC) -o$@ -c setup_.c + +setup_.c : $(SRCDIR)\setup.c + +translate$E $** > $@ + +$(OBJDIR)\sha1$O : sha1_.c sha1.h + $(TCC) -o$@ -c sha1_.c + +sha1_.c : $(SRCDIR)\sha1.c + +translate$E $** > $@ + +$(OBJDIR)\shun$O : shun_.c shun.h + $(TCC) -o$@ -c shun_.c + +shun_.c : $(SRCDIR)\shun.c + +translate$E $** > $@ + +$(OBJDIR)\skins$O : skins_.c skins.h + $(TCC) -o$@ -c skins_.c + +skins_.c : $(SRCDIR)\skins.c + +translate$E $** > $@ + +$(OBJDIR)\stat$O : stat_.c stat.h + $(TCC) -o$@ -c stat_.c + +stat_.c : $(SRCDIR)\stat.c + +translate$E $** > $@ + +$(OBJDIR)\style$O : style_.c style.h + $(TCC) -o$@ -c style_.c + +style_.c : $(SRCDIR)\style.c + +translate$E $** > $@ + +$(OBJDIR)\sync$O : sync_.c sync.h + $(TCC) -o$@ -c sync_.c + +sync_.c : $(SRCDIR)\sync.c + +translate$E $** > $@ + +$(OBJDIR)\tag$O : tag_.c tag.h + $(TCC) -o$@ -c tag_.c + +tag_.c : $(SRCDIR)\tag.c + +translate$E $** > $@ + +$(OBJDIR)\th_main$O : th_main_.c th_main.h + $(TCC) -o$@ -c th_main_.c + +th_main_.c : $(SRCDIR)\th_main.c + +translate$E $** > $@ + +$(OBJDIR)\timeline$O : timeline_.c timeline.h + $(TCC) -o$@ -c timeline_.c + +timeline_.c : $(SRCDIR)\timeline.c + +translate$E $** > $@ + +$(OBJDIR)\tkt$O : tkt_.c tkt.h + $(TCC) -o$@ -c tkt_.c + +tkt_.c : $(SRCDIR)\tkt.c + +translate$E $** > $@ + +$(OBJDIR)\tktsetup$O : tktsetup_.c tktsetup.h + $(TCC) -o$@ -c tktsetup_.c + +tktsetup_.c : $(SRCDIR)\tktsetup.c + +translate$E $** > $@ + +$(OBJDIR)\undo$O : undo_.c undo.h + $(TCC) -o$@ -c undo_.c + +undo_.c : $(SRCDIR)\undo.c + +translate$E $** > $@ + +$(OBJDIR)\update$O : update_.c update.h + $(TCC) -o$@ -c update_.c + +update_.c : $(SRCDIR)\update.c + +translate$E $** > $@ + +$(OBJDIR)\url$O : url_.c url.h + $(TCC) -o$@ -c url_.c + +url_.c : $(SRCDIR)\url.c + +translate$E $** > $@ + +$(OBJDIR)\user$O : user_.c user.h + $(TCC) -o$@ -c user_.c + +user_.c : $(SRCDIR)\user.c + +translate$E $** > $@ + +$(OBJDIR)\verify$O : verify_.c verify.h + $(TCC) -o$@ -c verify_.c + +verify_.c : $(SRCDIR)\verify.c + +translate$E $** > $@ + +$(OBJDIR)\vfile$O : vfile_.c vfile.h + $(TCC) -o$@ -c vfile_.c + +vfile_.c : $(SRCDIR)\vfile.c + +translate$E $** > $@ + +$(OBJDIR)\wiki$O : wiki_.c wiki.h + $(TCC) -o$@ -c wiki_.c + +wiki_.c : $(SRCDIR)\wiki.c + +translate$E $** > $@ + +$(OBJDIR)\wikiformat$O : wikiformat_.c wikiformat.h + $(TCC) -o$@ -c wikiformat_.c + +wikiformat_.c : $(SRCDIR)\wikiformat.c + +translate$E $** > $@ + +$(OBJDIR)\winhttp$O : winhttp_.c winhttp.h + $(TCC) -o$@ -c winhttp_.c + +winhttp_.c : $(SRCDIR)\winhttp.c + +translate$E $** > $@ + +$(OBJDIR)\xfer$O : xfer_.c xfer.h + $(TCC) -o$@ -c xfer_.c + +xfer_.c : $(SRCDIR)\xfer.c + +translate$E $** > $@ + +$(OBJDIR)\zip$O : zip_.c zip.h + $(TCC) -o$@ -c zip_.c + +zip_.c : $(SRCDIR)\zip.c + +translate$E $** > $@ + +headers: makeheaders$E page_index.h VERSION.h + +makeheaders$E add_.c:add.h allrepo_.c:allrepo.h attach_.c:attach.h bag_.c:bag.h blob_.c:blob.h branch_.c:branch.h browse_.c:browse.h captcha_.c:captcha.h cgi_.c:cgi.h checkin_.c:checkin.h checkout_.c:checkout.h clearsign_.c:clearsign.h clone_.c:clone.h comformat_.c:comformat.h configure_.c:configure.h content_.c:content.h db_.c:db.h delta_.c:delta.h deltacmd_.c:deltacmd.h descendants_.c:descendants.h diff_.c:diff.h diffcmd_.c:diffcmd.h doc_.c:doc.h encode_.c:encode.h event_.c:event.h file_.c:file.h finfo_.c:finfo.h graph_.c:graph.h http_.c:http.h http_socket_.c:http_socket.h http_ssl_.c:http_ssl.h http_transport_.c:http_transport.h info_.c:info.h login_.c:login.h main_.c:main.h manifest_.c:manifest.h md5_.c:md5.h merge_.c:merge.h merge3_.c:merge3.h name_.c:name.h pivot_.c:pivot.h popen_.c:popen.h pqueue_.c:pqueue.h printf_.c:printf.h rebuild_.c:rebuild.h report_.c:report.h rss_.c:rss.h schema_.c:schema.h search_.c:search.h setup_.c:setup.h sha1_.c:sha1.h shun_.c:shun.h skins_.c:skins.h stat_.c:stat.h style_.c:style.h sync_.c:sync.h tag_.c:tag.h th_main_.c:th_main.h timeline_.c:timeline.h tkt_.c:tkt.h tktsetup_.c:tktsetup.h undo_.c:undo.h update_.c:update.h url_.c:url.h user_.c:user.h verify_.c:verify.h vfile_.c:vfile.h wiki_.c:wiki.h wikiformat_.c:wikiformat.h winhttp_.c:winhttp.h xfer_.c:xfer.h zip_.c:zip.h $(SRCDIR)\sqlite3.h $(SRCDIR)\th.h VERSION.h + @copy /Y nul: headers ADDED win/Makefile.msc Index: win/Makefile.msc ================================================================== --- win/Makefile.msc +++ win/Makefile.msc @@ -0,0 +1,529 @@ +# DO NOT EDIT +# +# This file is automatically generated. Instead of editing this +# file, edit "makemake.tcl" then run +# "tclsh src/makemake.tcl msc > win/Makefile.msc" +# to regenerate this file. +B = .. +SRCDIR = $B\src +OBJDIR = . +O = .obj +E = .exe + +# Maybe MSCDIR, SSL, ZLIB, or INCL needs adjustment +MSCDIR = c:\msc + +# Uncomment below for SSL support +SSL = +SSLLIB = +#SSL = -DFOSSIL_ENABLE_SSL=1 +#SSLLIB = ssleay32.lib libeay32.lib user32.lib gdi32.lib advapi32.lib + +# zlib options +# When using precompiled from http://zlib.net/zlib125-dll.zip +#ZINCDIR = C:\zlib125-dll\include +#ZLIBDIR = C:\zlib125-dll\lib +#ZLIB = zdll.lib +ZINCDIR = $(MSCDIR)\extra\include +ZLIBDIR = $(MSCDIR)\extra\lib +ZLIB = zlib.lib + +INCL = -I. -I$(SRCDIR) -I$B\win\include -I$(MSCDIR)\extra\include -I$(ZINCDIR) + +MSCDEF = -Dstrncasecmp=memicmp -Dstrcasecmp=stricmp +I18N = -DFOSSIL_I18N=0 + +CFLAGS = -nologo -MT -O2 +BCC = $(CC) $(CFLAGS) +TCC = $(CC) -c $(CFLAGS) $(MSCDEF) $(I18N) $(SSL) $(INCL) +LIBS = $(ZLIB) ws2_32.lib $(SSLLIB) +LIBDIR = -LIBPATH:$(MSCDIR)\extra\lib -LIBPATH:$(ZLIBDIR) + +SRC = add_.c allrepo_.c attach_.c bag_.c blob_.c branch_.c browse_.c captcha_.c cgi_.c checkin_.c checkout_.c clearsign_.c clone_.c comformat_.c configure_.c content_.c db_.c delta_.c deltacmd_.c descendants_.c diff_.c diffcmd_.c doc_.c encode_.c file_.c finfo_.c graph_.c http_.c http_socket_.c http_ssl_.c http_transport_.c info_.c login_.c main_.c manifest_.c md5_.c merge_.c merge3_.c name_.c pivot_.c popen_.c pqueue_.c printf_.c rebuild_.c report_.c rss_.c schema_.c search_.c setup_.c sha1_.c shun_.c skins_.c stat_.c style_.c sync_.c tag_.c th_main_.c timeline_.c tkt_.c tktsetup_.c undo_.c update_.c url_.c user_.c verify_.c vfile_.c wiki_.c wikiformat_.c winhttp_.c xfer_.c zip_.c + +OBJ = $(OBJDIR)\add$O $(OBJDIR)\allrepo$O $(OBJDIR)\attach$O $(OBJDIR)\bag$O $(OBJDIR)\blob$O $(OBJDIR)\branch$O $(OBJDIR)\browse$O $(OBJDIR)\captcha$O $(OBJDIR)\cgi$O $(OBJDIR)\checkin$O $(OBJDIR)\checkout$O $(OBJDIR)\clearsign$O $(OBJDIR)\clone$O $(OBJDIR)\comformat$O $(OBJDIR)\configure$O $(OBJDIR)\content$O $(OBJDIR)\db$O $(OBJDIR)\delta$O $(OBJDIR)\deltacmd$O $(OBJDIR)\descendants$O $(OBJDIR)\diff$O $(OBJDIR)\diffcmd$O $(OBJDIR)\doc$O $(OBJDIR)\encode$O $(OBJDIR)\file$O $(OBJDIR)\finfo$O $(OBJDIR)\graph$O $(OBJDIR)\http$O $(OBJDIR)\http_socket$O $(OBJDIR)\http_ssl$O $(OBJDIR)\http_transport$O $(OBJDIR)\info$O $(OBJDIR)\login$O $(OBJDIR)\main$O $(OBJDIR)\manifest$O $(OBJDIR)\md5$O $(OBJDIR)\merge$O $(OBJDIR)\merge3$O $(OBJDIR)\name$O $(OBJDIR)\pivot$O $(OBJDIR)\popen$O $(OBJDIR)\pqueue$O $(OBJDIR)\printf$O $(OBJDIR)\rebuild$O $(OBJDIR)\report$O $(OBJDIR)\rss$O $(OBJDIR)\schema$O $(OBJDIR)\search$O $(OBJDIR)\setup$O $(OBJDIR)\sha1$O $(OBJDIR)\shun$O $(OBJDIR)\skins$O $(OBJDIR)\stat$O $(OBJDIR)\style$O $(OBJDIR)\sync$O $(OBJDIR)\tag$O $(OBJDIR)\th_main$O $(OBJDIR)\timeline$O $(OBJDIR)\tkt$O $(OBJDIR)\tktsetup$O $(OBJDIR)\undo$O $(OBJDIR)\update$O $(OBJDIR)\url$O $(OBJDIR)\user$O $(OBJDIR)\verify$O $(OBJDIR)\vfile$O $(OBJDIR)\wiki$O $(OBJDIR)\wikiformat$O $(OBJDIR)\winhttp$O $(OBJDIR)\xfer$O $(OBJDIR)\zip$O $(OBJDIR)\sqlite3$O $(OBJDIR)\th$O $(OBJDIR)\th_lang$O + + +APPNAME = $(OBJDIR)\fossil$(E) + +all: $(OBJDIR) $(APPNAME) + +$(APPNAME) : translate$E mkindex$E headers $(OBJ) $(OBJDIR)\linkopts + cd $(OBJDIR) + link -LINK -OUT:$@ $(LIBDIR) @linkopts + +$(OBJDIR)\linkopts: $B\win\Makefile.msc + echo add allrepo attach bag blob branch browse captcha cgi checkin checkout clearsign clone comformat configure content db delta deltacmd descendants diff diffcmd doc encode file finfo graph http http_socket http_ssl http_transport info login main manifest md5 merge merge3 name pivot popen pqueue printf rebuild report rss schema search setup sha1 shun skins stat style sync tag th_main timeline tkt tktsetup undo update url user verify vfile wiki wikiformat winhttp xfer zip sqlite3 th th_lang > $@ + echo $(LIBS) >> $@ + + + + +$(OBJDIR): + @-mkdir $@ + +translate$E: $(SRCDIR)\translate.c + $(BCC) $** + +makeheaders$E: $(SRCDIR)\makeheaders.c + $(BCC) $** + +mkindex$E: $(SRCDIR)\mkindex.c + $(BCC) $** + +version$E: $B\win\version.c + $(BCC) $** + +$(OBJDIR)\sqlite3$O : $(SRCDIR)\sqlite3.c + $(TCC) /Fo$@ -c -DSQLITE_OMIT_LOAD_EXTENSION=1 -DSQLITE_THREADSAFE=0 -DSQLITE_DEFAULT_FILE_FORMAT=4 -Dlocaltime=fossil_localtime -DSQLITE_ENABLE_LOCKING_STYLE=0 $** + +$(OBJDIR)\th$O : $(SRCDIR)\th.c + $(TCC) /Fo$@ -c $** + +$(OBJDIR)\th_lang$O : $(SRCDIR)\th_lang.c + $(TCC) /Fo$@ -c $** + +VERSION.h : version$E $B\manifest.uuid $B\manifest + $** > $@ + +page_index.h: mkindex$E $(SRC) + $** > $@ + +clean: + -del $(OBJDIR)\*.obj + -del *.obj *_.c *.h *.map + -del headers linkopts + +realclean: + -del $(APPNAME) translate$E mkindex$E makeheaders$E version$E + + +$(OBJDIR)\add$O : add_.c add.h + $(TCC) /Fo$@ -c add_.c + +add_.c : $(SRCDIR)\add.c + translate$E $** > $@ + +$(OBJDIR)\allrepo$O : allrepo_.c allrepo.h + $(TCC) /Fo$@ -c allrepo_.c + +allrepo_.c : $(SRCDIR)\allrepo.c + translate$E $** > $@ + +$(OBJDIR)\attach$O : attach_.c attach.h + $(TCC) /Fo$@ -c attach_.c + +attach_.c : $(SRCDIR)\attach.c + translate$E $** > $@ + +$(OBJDIR)\bag$O : bag_.c bag.h + $(TCC) /Fo$@ -c bag_.c + +bag_.c : $(SRCDIR)\bag.c + translate$E $** > $@ + +$(OBJDIR)\blob$O : blob_.c blob.h + $(TCC) /Fo$@ -c blob_.c + +blob_.c : $(SRCDIR)\blob.c + translate$E $** > $@ + +$(OBJDIR)\branch$O : branch_.c branch.h + $(TCC) /Fo$@ -c branch_.c + +branch_.c : $(SRCDIR)\branch.c + translate$E $** > $@ + +$(OBJDIR)\browse$O : browse_.c browse.h + $(TCC) /Fo$@ -c browse_.c + +browse_.c : $(SRCDIR)\browse.c + translate$E $** > $@ + +$(OBJDIR)\captcha$O : captcha_.c captcha.h + $(TCC) /Fo$@ -c captcha_.c + +captcha_.c : $(SRCDIR)\captcha.c + translate$E $** > $@ + +$(OBJDIR)\cgi$O : cgi_.c cgi.h + $(TCC) /Fo$@ -c cgi_.c + +cgi_.c : $(SRCDIR)\cgi.c + translate$E $** > $@ + +$(OBJDIR)\checkin$O : checkin_.c checkin.h + $(TCC) /Fo$@ -c checkin_.c + +checkin_.c : $(SRCDIR)\checkin.c + translate$E $** > $@ + +$(OBJDIR)\checkout$O : checkout_.c checkout.h + $(TCC) /Fo$@ -c checkout_.c + +checkout_.c : $(SRCDIR)\checkout.c + translate$E $** > $@ + +$(OBJDIR)\clearsign$O : clearsign_.c clearsign.h + $(TCC) /Fo$@ -c clearsign_.c + +clearsign_.c : $(SRCDIR)\clearsign.c + translate$E $** > $@ + +$(OBJDIR)\clone$O : clone_.c clone.h + $(TCC) /Fo$@ -c clone_.c + +clone_.c : $(SRCDIR)\clone.c + translate$E $** > $@ + +$(OBJDIR)\comformat$O : comformat_.c comformat.h + $(TCC) /Fo$@ -c comformat_.c + +comformat_.c : $(SRCDIR)\comformat.c + translate$E $** > $@ + +$(OBJDIR)\configure$O : configure_.c configure.h + $(TCC) /Fo$@ -c configure_.c + +configure_.c : $(SRCDIR)\configure.c + translate$E $** > $@ + +$(OBJDIR)\content$O : content_.c content.h + $(TCC) /Fo$@ -c content_.c + +content_.c : $(SRCDIR)\content.c + translate$E $** > $@ + +$(OBJDIR)\db$O : db_.c db.h + $(TCC) /Fo$@ -c db_.c + +db_.c : $(SRCDIR)\db.c + translate$E $** > $@ + +$(OBJDIR)\delta$O : delta_.c delta.h + $(TCC) /Fo$@ -c delta_.c + +delta_.c : $(SRCDIR)\delta.c + translate$E $** > $@ + +$(OBJDIR)\deltacmd$O : deltacmd_.c deltacmd.h + $(TCC) /Fo$@ -c deltacmd_.c + +deltacmd_.c : $(SRCDIR)\deltacmd.c + translate$E $** > $@ + +$(OBJDIR)\descendants$O : descendants_.c descendants.h + $(TCC) /Fo$@ -c descendants_.c + +descendants_.c : $(SRCDIR)\descendants.c + translate$E $** > $@ + +$(OBJDIR)\diff$O : diff_.c diff.h + $(TCC) /Fo$@ -c diff_.c + +diff_.c : $(SRCDIR)\diff.c + translate$E $** > $@ + +$(OBJDIR)\diffcmd$O : diffcmd_.c diffcmd.h + $(TCC) /Fo$@ -c diffcmd_.c + +diffcmd_.c : $(SRCDIR)\diffcmd.c + translate$E $** > $@ + +$(OBJDIR)\doc$O : doc_.c doc.h + $(TCC) /Fo$@ -c doc_.c + +doc_.c : $(SRCDIR)\doc.c + translate$E $** > $@ + +$(OBJDIR)\encode$O : encode_.c encode.h + $(TCC) /Fo$@ -c encode_.c + +encode_.c : $(SRCDIR)\encode.c + translate$E $** > $@ + +$(OBJDIR)\file$O : file_.c file.h + $(TCC) /Fo$@ -c file_.c + +file_.c : $(SRCDIR)\file.c + translate$E $** > $@ + +$(OBJDIR)\finfo$O : finfo_.c finfo.h + $(TCC) /Fo$@ -c finfo_.c + +finfo_.c : $(SRCDIR)\finfo.c + translate$E $** > $@ + +$(OBJDIR)\graph$O : graph_.c graph.h + $(TCC) /Fo$@ -c graph_.c + +graph_.c : $(SRCDIR)\graph.c + translate$E $** > $@ + +$(OBJDIR)\http$O : http_.c http.h + $(TCC) /Fo$@ -c http_.c + +http_.c : $(SRCDIR)\http.c + translate$E $** > $@ + +$(OBJDIR)\http_socket$O : http_socket_.c http_socket.h + $(TCC) /Fo$@ -c http_socket_.c + +http_socket_.c : $(SRCDIR)\http_socket.c + translate$E $** > $@ + +$(OBJDIR)\http_ssl$O : http_ssl_.c http_ssl.h + $(TCC) /Fo$@ -c http_ssl_.c + +http_ssl_.c : $(SRCDIR)\http_ssl.c + translate$E $** > $@ + +$(OBJDIR)\http_transport$O : http_transport_.c http_transport.h + $(TCC) /Fo$@ -c http_transport_.c + +http_transport_.c : $(SRCDIR)\http_transport.c + translate$E $** > $@ + +$(OBJDIR)\info$O : info_.c info.h + $(TCC) /Fo$@ -c info_.c + +info_.c : $(SRCDIR)\info.c + translate$E $** > $@ + +$(OBJDIR)\login$O : login_.c login.h + $(TCC) /Fo$@ -c login_.c + +login_.c : $(SRCDIR)\login.c + translate$E $** > $@ + +$(OBJDIR)\main$O : main_.c main.h + $(TCC) /Fo$@ -c main_.c + +main_.c : $(SRCDIR)\main.c + translate$E $** > $@ + +$(OBJDIR)\manifest$O : manifest_.c manifest.h + $(TCC) /Fo$@ -c manifest_.c + +manifest_.c : $(SRCDIR)\manifest.c + translate$E $** > $@ + +$(OBJDIR)\md5$O : md5_.c md5.h + $(TCC) /Fo$@ -c md5_.c + +md5_.c : $(SRCDIR)\md5.c + translate$E $** > $@ + +$(OBJDIR)\merge$O : merge_.c merge.h + $(TCC) /Fo$@ -c merge_.c + +merge_.c : $(SRCDIR)\merge.c + translate$E $** > $@ + +$(OBJDIR)\merge3$O : merge3_.c merge3.h + $(TCC) /Fo$@ -c merge3_.c + +merge3_.c : $(SRCDIR)\merge3.c + translate$E $** > $@ + +$(OBJDIR)\name$O : name_.c name.h + $(TCC) /Fo$@ -c name_.c + +name_.c : $(SRCDIR)\name.c + translate$E $** > $@ + +$(OBJDIR)\pivot$O : pivot_.c pivot.h + $(TCC) /Fo$@ -c pivot_.c + +pivot_.c : $(SRCDIR)\pivot.c + translate$E $** > $@ + +$(OBJDIR)\popen$O : popen_.c popen.h + $(TCC) /Fo$@ -c popen_.c + +popen_.c : $(SRCDIR)\popen.c + translate$E $** > $@ + +$(OBJDIR)\pqueue$O : pqueue_.c pqueue.h + $(TCC) /Fo$@ -c pqueue_.c + +pqueue_.c : $(SRCDIR)\pqueue.c + translate$E $** > $@ + +$(OBJDIR)\printf$O : printf_.c printf.h + $(TCC) /Fo$@ -c printf_.c + +printf_.c : $(SRCDIR)\printf.c + translate$E $** > $@ + +$(OBJDIR)\rebuild$O : rebuild_.c rebuild.h + $(TCC) /Fo$@ -c rebuild_.c + +rebuild_.c : $(SRCDIR)\rebuild.c + translate$E $** > $@ + +$(OBJDIR)\report$O : report_.c report.h + $(TCC) /Fo$@ -c report_.c + +report_.c : $(SRCDIR)\report.c + translate$E $** > $@ + +$(OBJDIR)\rss$O : rss_.c rss.h + $(TCC) /Fo$@ -c rss_.c + +rss_.c : $(SRCDIR)\rss.c + translate$E $** > $@ + +$(OBJDIR)\schema$O : schema_.c schema.h + $(TCC) /Fo$@ -c schema_.c + +schema_.c : $(SRCDIR)\schema.c + translate$E $** > $@ + +$(OBJDIR)\search$O : search_.c search.h + $(TCC) /Fo$@ -c search_.c + +search_.c : $(SRCDIR)\search.c + translate$E $** > $@ + +$(OBJDIR)\setup$O : setup_.c setup.h + $(TCC) /Fo$@ -c setup_.c + +setup_.c : $(SRCDIR)\setup.c + translate$E $** > $@ + +$(OBJDIR)\sha1$O : sha1_.c sha1.h + $(TCC) /Fo$@ -c sha1_.c + +sha1_.c : $(SRCDIR)\sha1.c + translate$E $** > $@ + +$(OBJDIR)\shun$O : shun_.c shun.h + $(TCC) /Fo$@ -c shun_.c + +shun_.c : $(SRCDIR)\shun.c + translate$E $** > $@ + +$(OBJDIR)\skins$O : skins_.c skins.h + $(TCC) /Fo$@ -c skins_.c + +skins_.c : $(SRCDIR)\skins.c + translate$E $** > $@ + +$(OBJDIR)\stat$O : stat_.c stat.h + $(TCC) /Fo$@ -c stat_.c + +stat_.c : $(SRCDIR)\stat.c + translate$E $** > $@ + +$(OBJDIR)\style$O : style_.c style.h + $(TCC) /Fo$@ -c style_.c + +style_.c : $(SRCDIR)\style.c + translate$E $** > $@ + +$(OBJDIR)\sync$O : sync_.c sync.h + $(TCC) /Fo$@ -c sync_.c + +sync_.c : $(SRCDIR)\sync.c + translate$E $** > $@ + +$(OBJDIR)\tag$O : tag_.c tag.h + $(TCC) /Fo$@ -c tag_.c + +tag_.c : $(SRCDIR)\tag.c + translate$E $** > $@ + +$(OBJDIR)\th_main$O : th_main_.c th_main.h + $(TCC) /Fo$@ -c th_main_.c + +th_main_.c : $(SRCDIR)\th_main.c + translate$E $** > $@ + +$(OBJDIR)\timeline$O : timeline_.c timeline.h + $(TCC) /Fo$@ -c timeline_.c + +timeline_.c : $(SRCDIR)\timeline.c + translate$E $** > $@ + +$(OBJDIR)\tkt$O : tkt_.c tkt.h + $(TCC) /Fo$@ -c tkt_.c + +tkt_.c : $(SRCDIR)\tkt.c + translate$E $** > $@ + +$(OBJDIR)\tktsetup$O : tktsetup_.c tktsetup.h + $(TCC) /Fo$@ -c tktsetup_.c + +tktsetup_.c : $(SRCDIR)\tktsetup.c + translate$E $** > $@ + +$(OBJDIR)\undo$O : undo_.c undo.h + $(TCC) /Fo$@ -c undo_.c + +undo_.c : $(SRCDIR)\undo.c + translate$E $** > $@ + +$(OBJDIR)\update$O : update_.c update.h + $(TCC) /Fo$@ -c update_.c + +update_.c : $(SRCDIR)\update.c + translate$E $** > $@ + +$(OBJDIR)\url$O : url_.c url.h + $(TCC) /Fo$@ -c url_.c + +url_.c : $(SRCDIR)\url.c + translate$E $** > $@ + +$(OBJDIR)\user$O : user_.c user.h + $(TCC) /Fo$@ -c user_.c + +user_.c : $(SRCDIR)\user.c + translate$E $** > $@ + +$(OBJDIR)\verify$O : verify_.c verify.h + $(TCC) /Fo$@ -c verify_.c + +verify_.c : $(SRCDIR)\verify.c + translate$E $** > $@ + +$(OBJDIR)\vfile$O : vfile_.c vfile.h + $(TCC) /Fo$@ -c vfile_.c + +vfile_.c : $(SRCDIR)\vfile.c + translate$E $** > $@ + +$(OBJDIR)\wiki$O : wiki_.c wiki.h + $(TCC) /Fo$@ -c wiki_.c + +wiki_.c : $(SRCDIR)\wiki.c + translate$E $** > $@ + +$(OBJDIR)\wikiformat$O : wikiformat_.c wikiformat.h + $(TCC) /Fo$@ -c wikiformat_.c + +wikiformat_.c : $(SRCDIR)\wikiformat.c + translate$E $** > $@ + +$(OBJDIR)\winhttp$O : winhttp_.c winhttp.h + $(TCC) /Fo$@ -c winhttp_.c + +winhttp_.c : $(SRCDIR)\winhttp.c + translate$E $** > $@ + +$(OBJDIR)\xfer$O : xfer_.c xfer.h + $(TCC) /Fo$@ -c xfer_.c + +xfer_.c : $(SRCDIR)\xfer.c + translate$E $** > $@ + +$(OBJDIR)\zip$O : zip_.c zip.h + $(TCC) /Fo$@ -c zip_.c + +zip_.c : $(SRCDIR)\zip.c + translate$E $** > $@ + +headers: makeheaders$E page_index.h VERSION.h + makeheaders$E add_.c:add.h allrepo_.c:allrepo.h attach_.c:attach.h bag_.c:bag.h blob_.c:blob.h branch_.c:branch.h browse_.c:browse.h captcha_.c:captcha.h cgi_.c:cgi.h checkin_.c:checkin.h checkout_.c:checkout.h clearsign_.c:clearsign.h clone_.c:clone.h comformat_.c:comformat.h configure_.c:configure.h content_.c:content.h db_.c:db.h delta_.c:delta.h deltacmd_.c:deltacmd.h descendants_.c:descendants.h diff_.c:diff.h diffcmd_.c:diffcmd.h doc_.c:doc.h encode_.c:encode.h file_.c:file.h finfo_.c:finfo.h graph_.c:graph.h http_.c:http.h http_socket_.c:http_socket.h http_ssl_.c:http_ssl.h http_transport_.c:http_transport.h info_.c:info.h login_.c:login.h main_.c:main.h manifest_.c:manifest.h md5_.c:md5.h merge_.c:merge.h merge3_.c:merge3.h name_.c:name.h pivot_.c:pivot.h popen_.c:popen.h pqueue_.c:pqueue.h printf_.c:printf.h rebuild_.c:rebuild.h report_.c:report.h rss_.c:rss.h schema_.c:schema.h search_.c:search.h setup_.c:setup.h sha1_.c:sha1.h shun_.c:shun.h skins_.c:skins.h stat_.c:stat.h style_.c:style.h sync_.c:sync.h tag_.c:tag.h th_main_.c:th_main.h timeline_.c:timeline.h tkt_.c:tkt.h tktsetup_.c:tktsetup.h undo_.c:undo.h update_.c:update.h url_.c:url.h user_.c:user.h verify_.c:verify.h vfile_.c:vfile.h wiki_.c:wiki.h wikiformat_.c:wikiformat.h winhttp_.c:winhttp.h xfer_.c:xfer.h zip_.c:zip.h $(SRCDIR)\sqlite3.h $(SRCDIR)\th.h VERSION.h + @copy /Y nul: headers ADDED win/fossil.ico Index: win/fossil.ico ================================================================== --- win/fossil.ico +++ win/fossil.ico cannot compute difference between binary files ADDED win/fossil.rc Index: win/fossil.rc ================================================================== --- win/fossil.rc +++ win/fossil.rc @@ -0,0 +1,38 @@ +#include +#include "VERSION.h" +#define _RC_COMPILE_ +#include "config.h" + +LANGUAGE LANG_ENGLISH,SUBLANG_ENGLISH_US + +VS_VERSION_INFO VERSIONINFO +FILEVERSION 1,0,0,0 +PRODUCTVERSION 1,0,0,0 +FILEFLAGSMASK 0x3F +FILEFLAGS 0x0 +FILEOS VOS__WINDOWS32 +FILETYPE VFT_APP +FILESUBTYPE VFT2_UNKNOWN +BEGIN + BLOCK "StringFileInfo" + BEGIN + BLOCK "040904B0" + BEGIN + VALUE "FileDescription", "distributed source code control system with integrated wiki and ticket-system\0" + VALUE "Comments", "compiler: "COMPILER_NAME"\0" + VALUE "FileVersion", MANIFEST_UUID"("COMPILER_NAME"\0" + VALUE "InternalName", "fossil\0" + VALUE "LegalCopyright", "Copyright (c) "MANIFEST_YEAR" D. Richard Hipp\0" + VALUE "OriginalFilename", "fossil.exe\0" + VALUE "ProductName", "fossil\0" + VALUE "ProductVersion", MANIFEST_VERSION" "MANIFEST_DATE" UTC\0" + END + END + BLOCK "VarFileInfo" + BEGIN + VALUE "Translation", 0x409, 0x4B0 + END +END + +8001 ICON "fossil.ico" + ADDED win/include/dirent.h Index: win/include/dirent.h ================================================================== --- win/include/dirent.h +++ win/include/dirent.h @@ -0,0 +1,230 @@ +/***************************************************************************** + * dirent.h - dirent API for Microsoft Visual Studio + * + * Copyright (C) 2006 Toni Ronkko + * + * Permission is hereby granted, free of charge, to any person obtaining + * a copy of this software and associated documentation files (the + * ``Software''), to deal in the Software without restriction, including + * without limitation the rights to use, copy, modify, merge, publish, + * distribute, sublicense, and/or sell copies of the Software, and to + * permit persons to whom the Software is furnished to do so, subject to + * the following conditions: + * + * The above copyright notice and this permission notice shall be included + * in all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED ``AS IS'', WITHOUT WARRANTY OF ANY KIND, EXPRESS + * OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. + * IN NO EVENT SHALL TONI RONKKO BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Dec 15, 2009, John Cunningham + * Added rewinddir member function + * + * Jan 18, 2008, Toni Ronkko + * Using FindFirstFileA and WIN32_FIND_DATAA to avoid converting string + * between multi-byte and unicode representations. This makes the + * code simpler and also allows the code to be compiled under MingW. Thanks + * to Azriel Fasten for the suggestion. + * + * Mar 4, 2007, Toni Ronkko + * Bug fix: due to the strncpy_s() function this file only compiled in + * Visual Studio 2005. Using the new string functions only when the + * compiler version allows. + * + * Nov 2, 2006, Toni Ronkko + * Major update: removed support for Watcom C, MS-DOS and Turbo C to + * simplify the file, updated the code to compile cleanly on Visual + * Studio 2005 with both unicode and multi-byte character strings, + * removed rewinddir() as it had a bug. + * + * Aug 20, 2006, Toni Ronkko + * Removed all remarks about MSVC 1.0, which is antiqued now. Simplified + * comments by removing SGML tags. + * + * May 14 2002, Toni Ronkko + * Embedded the function definitions directly to the header so that no + * source modules need to be included in the Visual Studio project. Removed + * all the dependencies to other projects so that this very header can be + * used independently. + * + * May 28 1998, Toni Ronkko + * First version. + *****************************************************************************/ +#ifndef DIRENT_H +#define DIRENT_H + +#include +#include +#include + + +typedef struct dirent +{ + char d_name[MAX_PATH + 1]; /* current dir entry (multi-byte char string) */ + WIN32_FIND_DATAA data; /* file attributes */ +} dirent; + + +typedef struct DIR +{ + dirent current; /* Current directory entry */ + int cached; /* Indicates un-processed entry in memory */ + HANDLE search_handle; /* File search handle */ + char patt[MAX_PATH + 3]; /* search pattern (3 = pattern + "\\*\0") */ +} DIR; + + +/* Forward declarations */ +static DIR *opendir (const char *dirname); +static struct dirent *readdir (DIR *dirp); +static int closedir (DIR *dirp); +static void rewinddir(DIR* dirp); + + +/* Use the new safe string functions introduced in Visual Studio 2005 */ +#if defined(_MSC_VER) && _MSC_VER >= 1400 +# define STRNCPY(dest,src,size) strncpy_s((dest),(size),(src),_TRUNCATE) +#else +# define STRNCPY(dest,src,size) strncpy((dest),(src),(size)) +#endif + + +/***************************************************************************** + * Open directory stream DIRNAME for read and return a pointer to the + * internal working area that is used to retrieve individual directory + * entries. + */ +static DIR *opendir(const char *dirname) +{ + DIR *dirp; + assert (dirname != NULL); + assert (strlen (dirname) < MAX_PATH); + + /* construct new DIR structure */ + dirp = (DIR*) malloc (sizeof (struct DIR)); + if (dirp != NULL) { + char *p; + + /* take directory name... */ + STRNCPY (dirp->patt, dirname, sizeof(dirp->patt)); + dirp->patt[MAX_PATH] = '\0'; + + /* ... and append search pattern to it */ + p = strchr (dirp->patt, '\0'); + if (dirp->patt < p && *(p-1) != '\\' && *(p-1) != ':') { + *p++ = '\\'; + } + *p++ = '*'; + *p = '\0'; + + /* open stream and retrieve first file */ + dirp->search_handle = FindFirstFileA (dirp->patt, &dirp->current.data); + if (dirp->search_handle == INVALID_HANDLE_VALUE) { + /* invalid search pattern? */ + free (dirp); + return NULL; + } + + /* there is an un-processed directory entry in memory now */ + dirp->cached = 1; + } + + return dirp; +} + + +/***************************************************************************** + * Read a directory entry, and return a pointer to a dirent structure + * containing the name of the entry in d_name field. Individual directory + * entries returned by this very function include regular files, + * sub-directories, pseudo-directories "." and "..", but also volume labels, + * hidden files and system files may be returned. + */ +static struct dirent *readdir(DIR *dirp) +{ + assert (dirp != NULL); + + if (dirp->search_handle == INVALID_HANDLE_VALUE) { + /* directory stream was opened/rewound incorrectly or ended normally */ + return NULL; + } + + /* get next directory entry */ + if (dirp->cached != 0) { + /* a valid directory entry already in memory */ + dirp->cached = 0; + } else { + /* read next directory entry from disk */ + if (FindNextFileA (dirp->search_handle, &dirp->current.data) == FALSE) { + /* the very last file has been processed or an error occured */ + FindClose (dirp->search_handle); + dirp->search_handle = INVALID_HANDLE_VALUE; + return NULL; + } + } + + /* copy as a multibyte character string */ + STRNCPY ( dirp->current.d_name, + dirp->current.data.cFileName, + sizeof(dirp->current.d_name) ); + dirp->current.d_name[MAX_PATH] = '\0'; + + return &dirp->current; +} + + +/***************************************************************************** + * Close directory stream opened by opendir() function. Close of the + * directory stream invalidates the DIR structure as well as any previously + * read directory entry. + */ +static int closedir(DIR *dirp) +{ + assert (dirp != NULL); + + /* release search handle */ + if (dirp->search_handle != INVALID_HANDLE_VALUE) { + FindClose (dirp->search_handle); + dirp->search_handle = INVALID_HANDLE_VALUE; + } + + /* release directory handle */ + free (dirp); + return 0; +} + + +/***************************************************************************** + * Resets the position of the directory stream to which dirp refers to the + * beginning of the directory. It also causes the directory stream to refer + * to the current state of the corresponding directory, as a call to opendir() + * would have done. If dirp does not refer to a directory stream, the effect + * is undefined. + */ +static void rewinddir(DIR* dirp) +{ + /* release search handle */ + if (dirp->search_handle != INVALID_HANDLE_VALUE) { + FindClose (dirp->search_handle); + dirp->search_handle = INVALID_HANDLE_VALUE; + } + + /* open new search handle and retrieve first file */ + dirp->search_handle = FindFirstFileA (dirp->patt, &dirp->current.data); + if (dirp->search_handle == INVALID_HANDLE_VALUE) { + /* invalid search pattern? */ + free (dirp); + return; + } + + /* there is an un-processed directory entry in memory now */ + dirp->cached = 1; +} + + +#endif /*DIRENT_H*/ ADDED win/include/unistd.h Index: win/include/unistd.h ================================================================== --- win/include/unistd.h +++ win/include/unistd.h @@ -0,0 +1,47 @@ +#ifndef _UNISTD_H +#define _UNISTD_H 1 + +/* This file intended to serve as a drop-in replacement for + * unistd.h on Windows + * Please add functionality as neeeded + */ + +#include +#include +#define srandom srand +#define random rand +#if defined(__DMC__) +#endif + +#if defined(_WIN32) +#define _CRT_SECURE_NO_WARNINGS 1 + +#ifndef F_OK +#define F_OK 0 +#endif /* not F_OK */ + +#ifndef X_OK +#define X_OK 1 +#endif /* not X_OK */ + +#ifndef R_OK +#define R_OK 2 +#endif /* not R_OK */ + +#ifndef W_OK +#define W_OK 4 +#endif /* not W_OK */ + +#define S_ISDIR(m) (((m) & S_IFMT) == S_IFDIR) +#define S_ISREG(m) (((m) & S_IFMT) == S_IFREG) + + + +#endif + +#define access _access +#define ftruncate _chsize + +#define ssize_t int + +#endif /* unistd.h */ ADDED win/version.c Index: win/version.c ================================================================== --- win/version.c +++ win/version.c @@ -0,0 +1,20 @@ +#include + +int main(int argc, char *argv[]){ + FILE *m,*u; + char b[10240]; + u = fopen(argv[1],"r"); + fgets(b, sizeof(b)-1,u); + b[strlen(b)-1] =0; + printf("#define MANIFEST_UUID \"%s\"\n",b); + printf("#define MANIFEST_VERSION \"[%10.10s]\"\n",b); + m = fopen(argv[2],"r"); + while(b == fgets(b, sizeof(b)-1,m)){ + if(0 == strncmp("D ",b,2)){ + printf("#define MANIFEST_DATE \"%.10s %.8s\"\n",b+2,b+13); + printf("#define MANIFEST_YEAR \"%.4s\"\n",b+2); + return 0; + } + } + return 1; +} Index: www/branching.wiki ================================================================== --- www/branching.wiki +++ www/branching.wiki @@ -155,10 +155,11 @@ accident that stems from concurrent development. In figure 4, giving check-in 2 multiple children is a deliberate act. So, to a good approximation, we define forking to be by accident and branching to be by intent. Apart from that, they are the same. +

Tags and properties are used in fossil to help express the intent, and thus to distinguish between forks and branches. Figure 5 shows the same scenario as figure 4 but with tags and properties added: ADDED www/event.wiki Index: www/event.wiki ================================================================== --- www/event.wiki +++ www/event.wiki @@ -0,0 +1,75 @@ +Events + +

+ +In Fossil, and "event" is a special kind of [./wikitheory.wiki | wiki page] +that is associated with a point in time rather than having a page name. +Each event causes a single entry to appear on the [/timeline | Timeline Page]. +Clicking on the hyperlink of the timeline entry cause a jump to the wiki +content for the event. The wiki content, the timeline entry text, the +time of the event, and the timeline background color can all be edited. + +As with check-ins, wiki, and tickets, all events automatically synchronize +to other repositories. Hence, events can be viewed, created, and edited +off-line. And the complete edit history for events is maintained +for auditing purposes. + +Possible uses for events include: + + * Milestones. Project milestones, such as releases or beta-test + cycles, can be recorded as events. The timeline entry for the event + can be something simple like "Version 1.2.3" perhaps with a bright + color background to draw attention to the entry and the wiki content + can contain release notes, for example. + + * Blog Entries. Blog entries from developers describing the current + state of a project, or rational for various design decisions, or + roadmaps for future development, can be entered as events. + + * Process Checkpoints. For projects that have a formal process, + events can be used to record the completion or the initiation of + various process steps. For example, an event can be used to record + the successful completion of a long-running test, perhaps with + performance results and details of where the test was run and who + ran it recorded in the wiki content. + + * News Articles. Significant occurrences in the lifecycle of + a project can be recorded as news articles using events. Perhaps the + domain name of the canonical website for a project changes, or new + server hardware is obtained. Such happenings are appropriate for + reporting as news. + + * Announcements. Changes to the composition of the development + team or acquisition of new project sponsors can be communicated as + announcements which can be implemented as events. + +No project is required to use events. But events can help many projects +stay better organized and provide a better historical record of the +development progress. + +

+ +Because events are considered a special kind of wiki, +users must have permission to read wiki in order read events. +Enable the "j" permission under the /Setup/Users menu in order +to give specific users or user classes the ability to view wiki +and events. + +Events show up on the timeline. Click on the hyperlink beside the +event title to see the details of the event. + +

+ +There is a hyperlink under the /Wiki menu that can be used to create +new events. And there is a submenu hyperlink on event displays for +editing existing events. + +Users must have check-in privileges (permission "i") in order to +create or edit events. In addition, users must have create-wiki +privilege (permission "f") to create new events and edit-wiki +privilege (permission "k") in order to edit existing events. + +If the first non-whitespace text of the event wiki content is +<title>...</title> then that markup is omitted from +the body of the wiki pages and is instead displayed as the page +title. Index: www/fileformat.wiki ================================================================== --- www/fileformat.wiki +++ www/fileformat.wiki @@ -43,13 +43,14 @@

[#cluster | Clusters]

[#ctrl | Control Artifacts]

[#wikichng | Wiki Pages]

[#tktchng | Ticket Changes]

[#attachment | Attachments]

[#event | Events]

-These five artifact types are described in the sequel. +These seven artifact types are described in the sequel. In the current implementation (as of 2009-01-25) the artifacts that make up a fossil repository are stored in in as delta- and zlib-compressed blobs in an SQLite database. This is an implementation detail and might change in a future release. For @@ -56,10 +57,13 @@ the purpose of this article "file format" means the format of the artifacts, not how the artifacts are stored on disk. It is the artifact format that is intended to be enduring. The specifics of how artifacts are stored on disk, though stable, is not intended to live as long as the artifact format. + +All of the artifacts can be extracted from a Fossil repository using +the "fossil deconstruct" command.

A manifest defines a check-in or version of the project @@ -91,26 +95,35 @@ may contain no additional text or data beyond what is described here. Allowed cards in the manifest are as follows:

+B baseline-manifest
C checkin-comment
D time-and-date-stamp
F filename SHA1-hash permissions old-name
P SHA1-hash+
R repository-checksum
T (+|-|*)tag-name * ?value?
U user-login
Z manifest-checksum

+ +A manifest may optionally have a single B-card. The B-card specifies +another manifest that serves as the "baseline" for this manifest. A +manifest that has a B-card is called a delta-manifest and a manifest +that omits the B-card is a baseline-manifest. The other manifest +identified by the argument of the B-card must be a baseline-manifest. +A baseline-manifest records the complete contents of a checkin. +A delta-manifest records only changes from its baseline. A manifest must have exactly one C-card. The sole argument to the C-card is a check-in comment that describes the check-in that the manifest defines. The check-in comment is text. The following escape sequences are applied to the text: A space (ASCII 0x20) is represented as "\s" (ASCII 0x5C, 0x73). A -newline (ASCII 0x0a) is "\n" (ASCII 0x6C, x6E). A backslash +newline (ASCII 0x0a) is "\n" (ASCII 0x5C, x6E). A backslash (ASCII 0x5C) is represented as two backslashes "\\". Apart from space and newline, no other whitespace characters are allowed in the check-in comment. Nor are any unprintable characters allowed in the comment. @@ -121,26 +134,29 @@

YYYY-MM-DDTHH:MM:SS

-A manifest has zero or more F-cards. Each F-card defines a file -(other than the manifest itself) which is part of the check-in that -the manifest defines. There are two, three, or four arguments. +A manifest has zero or more F-cards. Each F-card identifies a file +that is part of the check-in. There are one, two, three, or four arguments. The first argument is the pathname of the file in the check-in relative to the root of the project file hierarchy. No ".." or "." directories are allowed within the filename. Space characters are escaped as in C-card comment text. Backslash characters and newlines are not allowed within filenames. The directory separator character is a forward slash (ASCII 0x2F). The second argument to the F-card is the full 40-character lower-case hexadecimal SHA1 hash of the content -artifact. The optional 3rd argument defines any special access +artifact. The second argument is required for baseline manifests +but is optional for delta manifests. When the second argument to the +F-card is omitted, it means that the file has been deleted relative +to the baseline. The optional 3rd argument defines any special access permissions associated with the file. The only special code currently defined is "x" which means that the file is executable. All files are always readable and writable. This can be expressed by "w" permission -if desired but is optional. +if desired but is optional. The file format might be extended with +new permission letters in the future. The optional 4th argument is the name of the same file as it existed in the parent check-in. If the name of the file is unchanged from its parent, then the 4th argument is omitted. A manifest has zero or one P-cards. Most manifests have one P-card. @@ -165,12 +181,13 @@ repository, append a single space (ASCII 0x20), the size of the file in ASCII decimal, a single newline character (ASCII 0x0A), and the complete text of the file. Compute the MD5 checksum of the result. -A manifest might contain one or more T-cards used to set tags or -properties on the check-in. The format of the T-card is the same as +A manifest might contain one or more T-cards used to set +[./branching.wiki#tags | tags or properties] +on the check-in. The format of the T-card is the same as described in Control Artifacts section below, except that the second argument is the single characcter "*" instead of an artifact ID. The * in place of the artifact ID indicates that the tag or property applies to the current artifact. It is not possible to encode the current artifact ID as part of an artifact, @@ -182,11 +199,11 @@ Each manifest has a single U-card. The argument to the U-card is the login of the user who created the manifest. The login name is encoded using the same character escapes as is used for the check-in comment argument to the C-card. -A manifest has an option Z-card as its last line. The argument +A manifest has an optional Z-card as its last line. The argument to the Z-card is a 32-character lowercase hexadecimal MD5 hash of all prior lines of the manifest up to and including the newline character that immediately precedes the "Z". The Z-card is just a sanity check to prove that the manifest is well-formed and consistent. @@ -262,11 +279,12 @@ clearsigned. The D card and the Z card of a control artifact are the same as in a manifest. -The T card represents a "tag" or property that is applied to +The T card represents a [./branching.wiki#tags | tag or property] +that is applied to some other artifact. The T card has two or three values. The second argument is the 40 character lowercase artifact ID of the artifact to which the tag is to be applied. The first value is the tag name. The first character of the tag is either "+", "-", or "*". A "+" means the tag should be added @@ -324,11 +342,11 @@ of text in the wiki page. That text follows the newline character that terminates the W card. The wiki text is always followed by one extra newline. An example wiki artifact can be seen -[/artifact/7b2f5fd0e0 | here]. +[/artifact?name=7b2f5fd0e0&txt=1 | here].

A ticket-change artifact represents a change to a trouble ticket. @@ -375,25 +393,25 @@

An attachment artifact associates some other artifact that is the -attachment (the source artifact) with a ticket or wiki page to which +attachment (the source artifact) with a ticket or wiki page or event to which the attachment is connected (the target artifact). The following cards are allowed on an attachment artifact:

-A filename target ?source? -C comment
+A filename target ?source?
+C comment
D time-and-date-stamp
U user-name
Z checksum

The A card specifies a filename for the attachment in its first argument. The second argument to the A card is the name -of the wiki page or ticket to which the attachment is connected. The +of the wiki page or ticket or event to which the attachment is connected. The third argument is either missing or else it is the 40-character artifact ID of the attachment itself. A missing third argument means that the attachment should be deleted. The C card is an optional comment describing what the attachment is about. @@ -405,46 +423,121 @@ A single U card gives the name of the user to added the attachment. If an attachment is added anonymously, then the U card may be omitted. The Z card is the usual checksum over the rest of the attachment artifact. + + +

+ +An event artifact associates a timeline comment and a page of text +(similar to a wiki page) with a point in time. Events can be used +to record project milestones, release notes, blog entries, process +checkpoints, or news articles. +The following cards are allowed on an event artifact: + +

+C comment
+D time-and-date-stamp
+E event-time event-id
+P parent-artifact-id+
+T +tag-name * value
+U user-name
+W size \n text \n
+Z checksum +

+ +The C card contains text that is displayed on the timeline for the +event. Exactly one C card is required on an event artifact. + +A single D card is required to give the date and time when the +event artifact was created. This is different from the time at which +the event occurs. + +A single E card gives the time of the event (the point on the timeline +where the event is displayed) and a unique identifier for the event. +When there are multiple artifacts with the same event-id, the one with +the most recent D card is the only one used. The event-id must be a +40-character lower-case hexadecimal string. + +The option P card specifies a prior event with the same event-id from +which the current event is an edit. The P card is a hint to the system +that it might be space efficient to store one event as a delta of the +other. + +An event might contain one or more T-cards used to set +[./branching.wiki#tags | tags or properties] +on the event. The format of the T-card is the same as +described in [#ctrl | Control Artifacts] section above, except that the +second argument is the single characcter "*" instead of an +artifact ID and the name is always prefaced by "+". +The * in place of the artifact ID indicates that +the tag or property applies to the current artifact. It is not +possible to encode the current artifact ID as part of an artifact, +since the act of inserting the artifact ID would change the artifact ID, +hence a * is used to represent "self". The "+" on the +name means that tags can only be add and they can only be non-propagating +tags. A an event, T cards are normally used to set the background +display color for timelines. + +The optional U card gives name of the user who entered the event. + +A single W card provides wiki text for the document associated with the +event. The format of the W card is exactly the same as for a +[#wikichng | wiki artifact]. + +The Z card is the usual checksum over the rest of the attachment artifact. + -

The following table summaries the various kinds of cards that appear on Fossil artifacts: - + + + + + + + + + + + + + - + + @@ -451,14 +544,26 @@ + + + + + + + + + + + + @@ -469,18 +574,20 @@ + + @@ -487,15 +594,17 @@ + + @@ -505,14 +614,16 @@ + + @@ -522,15 +633,17 @@ + + @@ -540,16 +653,18 @@ + +

Card Format	Used By						Used By
Card Format	Manifest	Cluster	Control	Wiki	Ticket	Attachment	Event
A filename target source						X
B baseline	X
C coment-text	C comment-text	X					X	X
D date-time-stamp	X		X	X	X	X	X
E event-time event-id							X
F filename uuid permissions oldname	X									X
K ticket-uuid					X
L wiki-title				X
M uuid		X
		X
R md5sum	X								X		X
U username	X		X	X	X	X	X
		X			X
Z md5sum	X	X	X	X	X	X	X

Index: www/index.wiki ================================================================== --- www/index.wiki +++ www/index.wiki @@ -40,12 +40,13 @@ internet these days. What makes Fossil worthy of attention? 1. Bug Tracking And Wiki - In addition to doing [./concepts.wiki | distributed version control] like Git and Mercurial, - Fossil also supports [./bugtheory.wiki | distributed bug tracking] and - [./wikitheory.wiki | distributed wiki] all in a single + Fossil also supports [./bugtheory.wiki | distributed bug tracking], + [./wikitheory.wiki | distributed wiki], and a + [./event.wiki | distributed blog] mechanism all in a single integrated package. 2. Web Interface - Fossil has a built-in and easy-to-use [./webui.wiki | web interface] that simplifies project tracking and promotes situational awareness. @@ -88,11 +89,11 @@ 7. Robust & Reliable - Fossil stores content using an [./fileformat.wiki | enduring file format] in an SQLite database so that transactions are atomic even if interrupted by a power loss or system crash. Furthermore, automatic [./selfcheck.wiki | self-checks] verify that all aspects of - the repository are consistent prior to each commit. In nearly three years + the repository are consistent prior to each commit. In over three years of operation, no work has ever been lost after having been committed to a Fossil repository.

@@ -111,10 +112,12 @@ * A tutorial on [./branching.wiki | branching], what it means and how to do it using fossil. * The [./selfcheck.wiki | automatic self-check] mechanism helps insure project integrity. * Fossil contains a [./wikitheory.wiki | built-in wiki]. + * An [./event.wiki | Event] is a special kind of wiki page associated + with a point in time rather than a name. * There is a [http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users | mailing list] (with publicly readable [http://www.mail-archive.com/fossil-users@lists.fossil-scm.org | archives] available for discussing fossil issues. * [./stats.wiki | Performance statistics] taken from real-world projects Index: www/password.wiki ================================================================== --- www/password.wiki +++ www/password.wiki @@ -60,20 +60,20 @@ the web interface or direct SQL manipulation of the USER table. Note also that the password field is essentially ignored for the special users named "anonymous", "developer", "reader", and "nobody". It is not possible to authenticate as users "developer", "reader", or "nobody" and the authentication protocol -for "anonymous" use one-time captchas not persistent passwords. +for "anonymous" uses one-time captchas not persistent passwords.

When a user logs into Fossil using the web interface, the login name and password are sent in the clear to the server. The server then hashes the password and compares it against the value stored in USER.PW. If they match, the server sets a cookie on the client to record the login. This cookie contains a large amount of high-quality randomness -and is thus impossible to guess. The value of the cookie and the IP +and is thus intractable to guess. The value of the cookie and the IP address of the client is stored in the USER.COOKIE and USER.IPADDR fields of the USER table on the server. The USER.CEXPIRE field holds an expiration date for the cookie, encoded as a julian day number. On all subsequent HTTP requests, the cookie value is matched against the USER table to Index: www/qandc.wiki ================================================================== --- www/qandc.wiki +++ www/qandc.wiki @@ -63,11 +63,11 @@ Other projects are also adopting fossil. But fossil does not yet have the massive user base of git or mercurial. Fossil looks like the bug tracker that would be in your -Linksys Router's administration screen.
+Linksys Router's administration screen.

I take a pragmatic approach to software: form follows function. To me, it is more important to have a reliable, fast, efficient, enduring, and simple DVCS than one that looks pretty.
Index: www/quickstart.wiki ================================================================== --- www/quickstart.wiki +++ www/quickstart.wiki @@ -232,16 +232,19 @@
fossil ui repository-filename
-
The difference between these two command is that ui - attempts to automatically start your web browser pointing at the - server whereas server does not. +
The ui command is intended for accessing the web interface + from a local desktop. The ui command binds to the loopback IP + address only (and is thus makes the web interface visible only on the + local machine) and it automatically start your web browser pointing at the + server. For cross-machine collaboration, use the server command, + which binds on all IP addresses and does not try to start a web browser. You can omit the repository-filename if you are within - a checked-out local tree. This server uses port 8080 by default - but you can specify a different port using the -port command.
+ a checked-out local tree. The server uses port 8080 by default + but you can specify a different port using the -port option.

Command-line servers like this are useful when two people want to share a repository on temporary or ad-hoc basis. For a more permanent installation, you should use either the CGI server or the inetd server. Index: www/server.wiki ================================================================== --- www/server.wiki +++ www/server.wiki @@ -94,16 +94,20 @@
At this stage, the standalone server (e.g. "fossil server") does not support SSL.

- @
%s(zPrevDate)
+ @
%s(zPrevDate)
@

@ %s(zTime)

}else{ - @

} sqlite3_snprintf(sizeof(zShort), zShort, "%.10s", zUuid); sqlite3_snprintf(sizeof(zShortCkin), zShortCkin, "%.10s", zCkin); if( zUuid ){ if( g.okHistory ){ @@ -187,27 +187,29 @@ hyperlink_to_uuid(zShortCkin); @ %h(zCom) (user: hyperlink_to_user(zUser, zDate, ""); @ branch: %h(zBr)) if( g.okHistory && zUuid ){ + const char *z = zFilename; if( fpid ){ @ [diff] } - @ + @ @ [annotate] } - @

+
Various security concerns with hosted repositories
+ +
+
There are two main concerns relating to usage of Fossil for sharing sensitive information (source or any other data):

Interception of the Fossil synchronization stream, thereby capturing data, and -
Direct access to the Fossil repository on the server +
Direct access to the Fossil repository on the server +

Regarding the first, it is adequate to secure the server using SSL, and disallowing any non-SSL access. The data stream will be encrypted by the HTTPS protocol, rendering the data reasonably secure. The truly paranoid may wish to deploy ssh encrypted tunnels, but that is quite a bit more difficult and cumbersome to set up (particularly for a larger number of users).

Index: www/wikitheory.wiki ================================================================== --- www/wikitheory.wiki +++ www/wikitheory.wiki @@ -5,10 +5,11 @@ * Stand-alone wiki pages. * Description and comments in [./bugtheory.wiki | bug reports]. * Check-in comments. * [./embeddeddoc.wiki | Embedded documentation] files whose name ends in "wiki". + * [./event.wiki | Event descriptions]. The [/wiki_rules | formatting rules] for fossil wiki are designed to be simple and intuitive. The idea is that wiki provides paragraph breaks, numbered and bulleted lists, and hyperlinking for simple documents together with a safe subset of HTML for more complex

Event Time:	+ @ + @
Timeline Comment:	+ @ %h(zComment) + @
Background Color:	+ render_color_chooser(0, zClr, 0, "clr", "cclr"); + @
Tags:	+ @ + @
Page Content:	+ @ %h(zBody) + @
+ @ + @ + @ + @