Fossil

Check-in [cadfcba32c]
Login

Check-in [cadfcba32c]

Many hyperlinks are disabled.
Use anonymous login to enable hyperlinks.

Overview
Comment:Add a /robots.txt page - useful only when Fossil is deployed as a stand-alone server instead of as a sub-component to a larger website.
Downloads: Tarball | ZIP archive
Timelines: family | ancestors | descendants | both | trunk
Files: files | file ages | folders
SHA3-256: cadfcba32c7e20dedf8bb58e00eb15c3ac6c79eedb9017e5170aa90ba152b7ec
User & Date: drh 2024-08-15 14:11:10.319
References
2024-09-11
13:30
Update tests to account for new settings introduced with [1a0b304307] and [cadfcba32c]. ... (check-in: 6ead7d999e user: andybradford tags: trunk)
Context
2024-08-16
19:06
Update the built-in SQLite to the latest trunk version that includes the order-by-subquery optimization. ... (check-in: a8aaed4216 user: drh tags: trunk)
2024-08-15
14:11
Add a /robots.txt page - useful only when Fossil is deployed as a stand-alone server instead of as a sub-component to a larger website. ... (check-in: cadfcba32c user: drh tags: trunk)
13:02
Always apply the robot-restrict setting to self-declared robots. ... (check-in: 398ea9e425 user: drh tags: trunk)
Changes
Unified Diff Ignore Whitespace Patch
Changes to src/style.c.
1802
1803
1804
1805
1806
1807
1808











































** 'error' and can be styled via a (noscript > .error) CSS selector.
*/
void style_emit_noscript_for_js_page(void){
  CX("<noscript><div class='error'>"
     "This page requires JavaScript (ES2015, a.k.a. ES6, or newer)."
     "</div></noscript>");
}


















































>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
** 'error' and can be styled via a (noscript > .error) CSS selector.
*/
void style_emit_noscript_for_js_page(void){
  CX("<noscript><div class='error'>"
     "This page requires JavaScript (ES2015, a.k.a. ES6, or newer)."
     "</div></noscript>");
}

/*
** SETTING: robots-txt width=70 block-text keep-empty
**
** This setting is the override value for the /robots.txt file that
** Fossil returns when run as a stand-alone server for a domain.  As
** Fossil is seldom run as a stand-alone server (and is more commonly
** deployed as a CGI or SCGI or behind a reverse proxy) this setting
** rarely needed.  A reasonable default robots.txt is sent if this
** setting is empty.
*/

/*
** WEBPAGE: robots.txt
**
** Return text/plain which is the content of the "robots-txt" setting, if
** such a setting exists and is non-empty.  Or construct an RFC-9309 complaint
** robots.txt file and return that if there is not "robots.txt" setting.
**
** This is useful for robot exclusion in cases where Fossil is run as a
** stand-alone server in its own domain.  For the more common case where
** Fossil is run as a CGI, or SCGI, or a server that responding to a reverse
** proxy, the returns robots.txt file will not be at the top level of the
** domain, and so it will be pointless.
*/
void robotstxt_page(void){
  const char *z;
  static const char *zDflt = 
     "User-agent: *\n"
     "Allow: /doc\n"
     "Allow: /home\n"
     "Allow: /forum\n"
     "Allow: /technote\n"
     "Allow: /tktview\n"
     "Allow: /wiki\n"
     "Allow: /uv/\n"
     "Allow: /$\n"
     "Disallow: /*\n"
  ;
  z = db_get("robots-txt",zDflt);
  cgi_set_content_type("text/plain");
  cgi_append_content(z, -1);
}