Fossil

Check-in [778efb30f7]
Login

Check-in [778efb30f7]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Teach /chat search to use the same tokenizer as the main search index, defaulting to porter if the main search index is off, and reindex chat if the tokenizer is changed. The search config should arguably be expanded to provide the option of disabling chat search altogether, but that is beyond today's ambitions. Minor search result layout tweaks but there are still some fixes to do there.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | fts5-chat-search
Files: files | file ages | folders
SHA3-256: 778efb30f75a1804005562e3815baa70a40deaacef6af739ac97a5bae9e3575d
User & Date: stephan 2024-07-01 17:25:24.618
Context
2024-07-01
18:36
Ensure that chat search results get placed in the proper parent DOM element (this fixes some layout unsightliness). Remove the DOM elements related to loading more search result context when they have no more results to load. ... (check-in: 2e8f1a9a15 user: stephan tags: fts5-chat-search)
17:25
Teach /chat search to use the same tokenizer as the main search index, defaulting to porter if the main search index is off, and reindex chat if the tokenizer is changed. The search config should arguably be expanded to provide the option of disabling chat search altogether, but that is beyond today's ambitions. Minor search result layout tweaks but there are still some fixes to do there. ... (check-in: 778efb30f7 user: stephan tags: fts5-chat-search)
15:49
Do not filter out the chat-robot messages from /chat search, per feedback. ... (check-in: b5281f4e61 user: stephan tags: fts5-chat-search)
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/chat.c.
281
282
283
284
285
286
287





























288
289

290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
@   mdel INT,         -- msgid of another message to delete
@   file  BLOB        -- Text of the uploaded file, or NULL
@ );
;


/*





























** Make sure the repository data tables used by chat exist.  Create them
** if they do not.

*/
static void chat_create_tables(void){
  if( !db_table_exists("repository","chat") ){
    db_multi_exec(zChatSchema1/*works-like:""*/);
  }else if( !db_table_has_column("repository","chat","lmtime") ){
    if( !db_table_has_column("repository","chat","mdel") ){
      db_multi_exec("ALTER TABLE chat ADD COLUMN mdel INT");
    }
    db_multi_exec("ALTER TABLE chat ADD COLUMN lmtime TEXT");
  }

  if( !db_table_exists("repository", "chatfts1") ){
    db_multi_exec(
      "CREATE VIRTUAL TABLE repository.chatfts1 USING fts5("
      "    xmsg, content=chat, content_rowid=msgid, tokenize=porter"
      ");"
      "INSERT INTO repository.chatfts1(chatfts1) VALUES('rebuild');"
    );
  }
  db_multi_exec(
    "DROP TRIGGER IF EXISTS chat_ai;"
    "DROP TRIGGER IF EXISTS chat_ad;"
    "CREATE TEMP TRIGGER chat_ai AFTER INSERT ON chat BEGIN "
    "  INSERT INTO chatfts1(rowid, xmsg) VALUES(new.msgid, new.xmsg);"
    "END;"
    "CREATE TEMP TRIGGER chat_ad AFTER DELETE ON chat BEGIN "
    "  INSERT INTO chatfts1(chatfts1, rowid, xmsg) "
    "    VALUES('delete', old.msgid, old.xmsg);"
    "END;"
  );
}

/*







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
|
|
>










|
<
|
<
<
<
<
<
<
<
<
<
|


|







281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330

331









332
333
334
335
336
337
338
339
340
341
342
@   mdel INT,         -- msgid of another message to delete
@   file  BLOB        -- Text of the uploaded file, or NULL
@ );
;


/*
** Create or rebuild the /chat search index. Requires that the
** repository.chat table exists. If bForce is true, it will drop the
** chatfts1 table and recreate/reindex it. If bForce is 0, it will
** only index the chat content if the chatfts1 table does not already
** exist.
*/
void chat_rebuild_index(int bForce){
  if( bForce!=0 ){
    db_multi_exec("DROP TABLE IF EXISTS chatfts1");
  }
  if( bForce!=0 || !db_table_exists("repository", "chatfts1") ){
    const int tokType = search_tokenizer_type(0);
    const char *zTokenizer = search_tokenize_arg_for_type(
      tokType==FTS5TOK_NONE ? FTS5TOK_PORTER : tokType
      /* Special case: if fts search is disabled for the main repo
      ** content, use a default tokenizer here. */
    );
    assert( zTokenizer && zTokenizer[0] );
    db_multi_exec(
      "CREATE VIRTUAL TABLE repository.chatfts1 USING fts5("
      "    xmsg, content=chat, content_rowid=msgid%s"
      ");"
      "INSERT INTO repository.chatfts1(chatfts1) VALUES('rebuild');",
      zTokenizer/*safe-for-%s*/
    );
  }
}

/*
** Make sure the repository data tables used by chat exist.  Create
** them if they do not. Set up TEMP triggers (if needed) to update the
** chatfts1 table as the chat table is updated.
*/
static void chat_create_tables(void){
  if( !db_table_exists("repository","chat") ){
    db_multi_exec(zChatSchema1/*works-like:""*/);
  }else if( !db_table_has_column("repository","chat","lmtime") ){
    if( !db_table_has_column("repository","chat","mdel") ){
      db_multi_exec("ALTER TABLE chat ADD COLUMN mdel INT");
    }
    db_multi_exec("ALTER TABLE chat ADD COLUMN lmtime TEXT");
  }
  chat_rebuild_index(0);

  db_multi_exec(









    "CREATE TEMP TRIGGER IF NOT EXISTS chat_ai AFTER INSERT ON chat BEGIN "
    "  INSERT INTO chatfts1(rowid, xmsg) VALUES(new.msgid, new.xmsg);"
    "END;"
    "CREATE TEMP TRIGGER IF NOT EXISTS chat_ad AFTER DELETE ON chat BEGIN "
    "  INSERT INTO chatfts1(chatfts1, rowid, xmsg) "
    "    VALUES('delete', old.msgid, old.xmsg);"
    "END;"
  );
}

/*
Changes to src/search.c.
1635
1636
1637
1638
1639
1640
1641

1642
1643
1644
1645
1646
1647
1648
@ CREATE VIRTUAL TABLE IF NOT EXISTS repository.ftsidx
@   USING fts5(content="ftscontent", title, body%s);
;
static const char zFtsDrop[] =
@ DROP TABLE IF EXISTS repository.ftsidx;
@ DROP VIEW IF EXISTS repository.ftscontent;
@ DROP TABLE IF EXISTS repository.ftsdocs;

;

#if INTERFACE
/*
** Values for the search-tokenizer config option.
*/
#define FTS5TOK_NONE      0 /* disabled */







>







1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
@ CREATE VIRTUAL TABLE IF NOT EXISTS repository.ftsidx
@   USING fts5(content="ftscontent", title, body%s);
;
static const char zFtsDrop[] =
@ DROP TABLE IF EXISTS repository.ftsidx;
@ DROP VIEW IF EXISTS repository.ftscontent;
@ DROP TABLE IF EXISTS repository.ftsdocs;
@ DROP TABLE IF EXISTS repository.chatfts1;
;

#if INTERFACE
/*
** Values for the search-tokenizer config option.
*/
#define FTS5TOK_NONE      0 /* disabled */
1679
1680
1681
1682
1683
1684
1685















1686
1687
1688
1689
1690
1691
1692
    iFtsTokenizer = FTS5TOK_TRIGRAM;
  }else{
    iFtsTokenizer = is_truth(z) ? FTS5TOK_PORTER : FTS5TOK_NONE;
  }
  fossil_free(z);
  return iFtsTokenizer;
}
















/*
** Returns a string value suitable for use as the search-tokenizer
** setting's value, depending on the value of z. If z is 0 then the
** current search-tokenizer value is used as the basis for formulating
** the result (which may differ from the current value but will have
** the same meaning). Any unknown/unsupported value is interpreted as







>
>
>
>
>
>
>
>
>
>
>
>
>
>
>







1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
    iFtsTokenizer = FTS5TOK_TRIGRAM;
  }else{
    iFtsTokenizer = is_truth(z) ? FTS5TOK_PORTER : FTS5TOK_NONE;
  }
  fossil_free(z);
  return iFtsTokenizer;
}

/*
** Returns a string in the form ",tokenize=X", where X is the string
** counterpart of the given FTS5TOK_xyz value.  Returns "" if tokType
** does not correspond to a known FTS5 tokenizer.
*/
const char * search_tokenize_arg_for_type(int tokType){
  switch( tokType ){
    case FTS5TOK_PORTER: return ",tokenize=porter";
    case FTS5TOK_UNICODE61: return ",tokenize=unicode61";
    case FTS5TOK_TRIGRAM: return ",tokenize=trigram";
    case FTS5TOK_NONE:
    default: return "";
  }
}

/*
** Returns a string value suitable for use as the search-tokenizer
** setting's value, depending on the value of z. If z is 0 then the
** current search-tokenizer value is used as the basis for formulating
** the result (which may differ from the current value but will have
** the same meaning). Any unknown/unsupported value is interpreted as
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
}

/*
** Create or drop the tables associated with a full-text index.
*/
static int searchIdxExists = -1;
void search_create_index(void){
  const int useTokenizer = search_tokenizer_type(0);
  const char *zExtra;
  switch(useTokenizer){
    case FTS5TOK_PORTER: zExtra = ",tokenize=porter"; break;
    case FTS5TOK_UNICODE61: zExtra = ",tokenize=unicode61"; break;
    case FTS5TOK_TRIGRAM: zExtra = ",tokenize=trigram"; break;
    default: zExtra = ""; break;
  }
  search_sql_setup(g.db);
  db_multi_exec(zFtsSchema/*works-like:"%s"*/, zExtra/*safe-for-%s*/);
  searchIdxExists = 1;
}
void search_drop_index(void){
  db_multi_exec(zFtsDrop/*works-like:""*/);
  searchIdxExists = 0;







<
|
|
<
<
<
|
<







1740
1741
1742
1743
1744
1745
1746

1747
1748



1749

1750
1751
1752
1753
1754
1755
1756
}

/*
** Create or drop the tables associated with a full-text index.
*/
static int searchIdxExists = -1;
void search_create_index(void){

  const char *zExtra =
    search_tokenize_arg_for_type(search_tokenizer_type(0));



  assert( zExtra );

  search_sql_setup(g.db);
  db_multi_exec(zFtsSchema/*works-like:"%s"*/, zExtra/*safe-for-%s*/);
  searchIdxExists = 1;
}
void search_drop_index(void){
  db_multi_exec(zFtsDrop/*works-like:""*/);
  searchIdxExists = 0;
2055
2056
2057
2058
2059
2060
2061



2062
2063
2064
2065
2066
2067
2068
*/
void search_rebuild_index(void){
  fossil_print("rebuilding the search index...");
  fflush(stdout);
  search_create_index();
  search_fill_index();
  search_update_index(search_restrict(SRCH_ALL));



  fossil_print(" done\n");
}

/*
** COMMAND: fts-config*
**
** Usage: fossil fts-config ?SUBCOMMAND? ?ARGUMENT?







>
>
>







2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
*/
void search_rebuild_index(void){
  fossil_print("rebuilding the search index...");
  fflush(stdout);
  search_create_index();
  search_fill_index();
  search_update_index(search_restrict(SRCH_ALL));
  if( db_table_exists("repository","chat") ){
    chat_rebuild_index(1);
  }
  fossil_print(" done\n");
}

/*
** COMMAND: fts-config*
**
** Usage: fossil fts-config ?SUBCOMMAND? ?ARGUMENT?
Changes to src/style.chat.css.
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
  display: none;
}
body.chat #chat-user-list .chat-user.selected {
  font-weight: bold;
  text-decoration: underline;
}

body.cpage-chat-search .searchForm {
  margin-top: 1em;
}
body.cpage-chat-search .spacer-widget button {
  margin-left: 1ex;
  margin-right: 1ex;
  display: block;
}

body.chat .spacer-widget-buttons .up {
  margin-top: 0.5em;
  margin-bottom: 1em;
}
body.chat .spacer-widget-buttons .down {
  margin-top: 1em;
  margin-bottom: 0.5em;
}
body.chat .spacer-widget-buttons .all {
  margin-bottom: 0.75em;
}


body.chat .anim-rotate-360 {
  animation: rotate-360 750ms linear;
}
@keyframes rotate-360 {
  from { transform: rotate(0deg); }
  to { transform: rotate(360deg); }







|


|



<
<
<

<
<
<
<


<
<
<
<







611
612
613
614
615
616
617
618
619
620
621
622
623
624



625




626
627




628
629
630
631
632
633
634
  display: none;
}
body.chat #chat-user-list .chat-user.selected {
  font-weight: bold;
  text-decoration: underline;
}

body.chat .searchForm {
  margin-top: 1em;
}
body.chat .spacer-widget button {
  margin-left: 1ex;
  margin-right: 1ex;
  display: block;



  margin-top: 0.5em;




  margin-bottom: 0.5em;
}





body.chat .anim-rotate-360 {
  animation: rotate-360 750ms linear;
}
@keyframes rotate-360 {
  from { transform: rotate(0deg); }
  to { transform: rotate(360deg); }