Improve documentation and error messages for CGI

Modify example used to investigate #529
Document CGI will not work for systems without fork (#584)

docs/UserManual.md

@@ -76,10 +76,12 @@ CivetWeb can also be used to modify `.htpasswd` passwords files:
 
     CivetWeb -A <htpasswd_file> <realm> <user> <passwd>
 
-Unlike other web servers, CivetWeb does not require CGI scripts to be located
-in a special directory. CGI scripts can be anywhere. CGI (and SSI) files are
-recognized by the file name pattern. CivetWeb uses shell-like glob
-patterns. Pattern match starts at the beginning of the string, so essentially
+
+# Pattern matching
+
+CivetWeb uses shell-like glob patterns for several configuration options,
+e.g., CGI, SSI and Lua script files are recognized by the file name pattern. 
+Pattern match starts at the beginning of the string, so essentially
 patterns are prefix patterns. Syntax is as follows:
 
      **      Matches everything
@@ -94,6 +96,7 @@ All other characters in the pattern match themselves. Examples:
     /foo         Any string that begins with /foo
     **a$|**b$    Any string that ends with a or b
 
+
 # Configuration Options
 
 Below is a list of configuration options understood by CivetWeb.
@@ -841,6 +844,41 @@ An example is shown in
 [websocket.lua](https://github.com/civetweb/civetweb/blob/master/test/websocket.lua).
 
 
+# Using CGI
+
+Unlike some other web servers, CivetWeb does not require CGI scripts to be located
+in a special directory. CGI scripts files are recognized by the file name pattern
+and can be anywhere.
+
+When using CGI, make sure your CGI file names match the `cgi\_pattern` parameter
+configured for the server.
+Furthermore, you must either configure a `cgi\_interpreter` to be used for all
+CGI scripts, or all scripts must start with `#!` followed by the CGI
+interpreter executable, e.g.: `#!/path/to/perl.exe` or `#!/bin/sh`.
+
+See `cgi\_pattern` and `cgi\_interpreter` for more details.
+
+It is possible to disable CGI completely by building the server with
+the `NO\_CGI` define. Setting this define is required for operating 
+systems not supporting `fork/exec` or `CreateProcess` (since CGI is
+based on creating child processes, it will not be available on such
+operating systems for principle reasons).
+
+Every CGI request will spawn a new child process. Data sent from the
+HTTP client to the server is passed to stdin of the child process,
+while data written to stdout by the child process is sent back to the
+HTTP client.
+
+In case a CGI script cannot handle a particular request, it might
+write a short error message to stderr instead of writing to stdout.
+This error message is added to the server error log.
+
+A script should not write to stderr after writing a reply header
+to stdout. In case CGI libraries are writing to stderr (e.g., for
+logging/debugging), the CGI script should redirect stderr to a 
+user defined log file at the beginning of the script.
+
+
 # Common Problems
 - PHP doesn't work - getting empty page, or 'File not found' error. The
   reason for that is wrong paths to the interpreter. Remember that with PHP,
@@ -858,7 +896,7 @@ An example is shown in
 
 - CivetWeb fails to start. If CivetWeb exits immediately when started, this
   usually indicates a syntax error in the configuration file
-  (named `CivetWeb.conf` by default) or the command-line arguments.
+  (named `civetweb.conf` by default) or the command-line arguments.
   Syntax checking is omitted from CivetWeb to keep its size low. However,
   the Manual should be of help. Note: the syntax changes from time to time,
   so updating the config file might be necessary after executable update.

src/civetweb.c

@@ -10321,7 +10321,7 @@ handle_cgi_request(struct mg_connection *conn, const char *prog)
 	}
 
 	buf = NULL;
-	buflen = 16384;
+	buflen = conn->phys_ctx->max_request_size;
 	i = prepare_cgi_environment(conn, prog, &blk);
 	if (i != 0) {
 		blk.buf = NULL;
@@ -10432,7 +10432,7 @@ handle_cgi_request(struct mg_connection *conn, const char *prog)
 		                status);
 		mg_send_http_error(conn,
 		                   500,
-		                   "Error: CGI can not open fdout\nfopen: %s",
+		                   "Error: CGI can not open fderr\nfopen: %s",
 		                   status);
 		goto done;
 	}
@@ -10492,20 +10492,22 @@ handle_cgi_request(struct mg_connection *conn, const char *prog)
 		 * stderr. */
 		i = pull_all(err, conn, buf, (int)buflen);
 		if (i > 0) {
+			/* CGI program explicitly sent an error */
+			/* Write the error message to the internal log */
 			mg_cry_internal(conn,
 			                "Error: CGI program \"%s\" sent error "
 			                "message: [%.*s]",
 			                prog,
 			                i,
 			                buf);
+			/* Don't send the error message back to the client */
 			mg_send_http_error(conn,
 			                   500,
-			                   "Error: CGI program \"%s\" sent error "
-			                   "message: [%.*s]",
-			                   prog,
-			                   i,
-			                   buf);
+			                   "Error: CGI program \"%s\" failed.",
+			                   prog);
 		} else {
+			/* CGI program did not explicitly send an error, but a broken
+			 * respon header */
 			mg_cry_internal(conn,
 			                "Error: CGI program sent malformed or too big "
 			                "(>%u bytes) HTTP headers: [%.*s]",
@@ -10522,6 +10524,7 @@ handle_cgi_request(struct mg_connection *conn, const char *prog)
 			                   buf);
 		}
 
+		/* in both cases, abort processing CGI */
 		goto done;
 	}
 
@@ -10568,8 +10571,8 @@ handle_cgi_request(struct mg_connection *conn, const char *prog)
 	mg_write(conn, buf + headers_len, (size_t)(data_len - headers_len));
 
 	/* Read the rest of CGI output and send to the client */
+	DEBUG_TRACE("CGI: %s", "forward all data");
 	send_file_data(conn, &fout, 0, INT64_MAX);
-
 	DEBUG_TRACE("CGI: %s", "all data sent");
 
 done:

test/linux_stderr.cgi

@@ -1,10 +1,16 @@
-#!/bin/sh
+#!/bin/bash
 
 printf "Content-Type: text/plain\r\n"
 printf "\r\n"
 
-echo "This is a shell script called by CGI:"
-tree / 1>&2
-echo
-set
+echo "This is a shell script called by CGI"
+
+for ((number=1;number < 5000;number++))
+{
+echo "Write $number to stdout"
+echo "Write $number to stderr" 1>&2
+}
+
+echo "done"
+