FramerD Web Applications Ken Haase

Quick ref: [Generating Markup]  [HTML builtins]  [Making links]  [Complex Links]  [Dyamic Pages from FDScript]  [Script Variables]  [Using HTML forms]  [CGI-VAR and CGI-INIT]  [Cookies]  [Frames to Anchors]  [HTTP generation: connection control and metadata] [Using mod_fdserv] [Configuring your web server]

At its creation in the early 1990's, the World Wide Web was imagined as `just' a medium for global publication and conversation. It subsequently grew into (among other things) a framework for application deployment where the ubiquitous web browser provided a GUI for networked applications. FramerD and FDScript have been especially extended to take advantage of this environment in two different ways.

First, FDScript provides numerous special forms and operations for generating HTML --- the language interpreted by web browsers --- and XML, HTML's younger, more refined, and more versatile cousin. These mechanisms rely on an analogy between the recursive, nested structures of HTML/XML and the similar structure of Lisp (FDScript) expressions. This (in our unbiased opinion) makes it much easier to write programs which produce correct HTML (or XML) markup. The framework for generating HTML also contains special provisions for presenting FramerD frames as links to browse scripts which can present the frame to the user as an HTML document.

Second, FDScript programs are able to use the CGI protocol to gain access to particular information provided by clients to web servers. The CGI (for Common Gateway Interface) protocol was designed to allow browsers to pass along user information, especially the results of filling out online forms. FramerD scripts running on a web server can read the values which users put in forms and customize action and presentation based on these specifications.

Together, these two features make for a complete application framework, where FramerD applications can be written which are usable by any individual with a web browser. This document describes how to build such applications, starting with how FDScript can generate HTML, moving on to how it can access CGI parameters, and wrapping up with a discussion of how to get all of this working together under the Apache web server.

Generating Markup

Web documents are typically encoded and transmitted in a markup language where content is interspersed with machine-recognizable tags indicating document structure or presentation information. FDScript provides ways to automatically generate documents with markup information for two pervasive markup standards: XML and HTML. XML is an extensible standard for document markup; HTML is a more limited standard for mostly "display markup" which is a cornerstone of documents published on the World Wide Web. These standards are closely related and --- except for a few handfuls of exceptions --- HTML can be interpreted as a kind of XML.

The FDScript modules XMLGEN and HTMLGEN provide functions for generating markup based on the PRINTOUT model for producing formatted output. The PRINTOUT model uses Scheme expressions to generate complex documents. Expressions are evaluated depth-first from left to right, producing output; expressions which evaluate to strings are displayed verbatim; expressions evaluating to other (non-void) values are displayed with WRITE; and expressions which don't return values or return the void value are ignored (and are assumed to have done their own output). This allows new formatting directives to be implemented simply as functions which do output and return void values.

The most general primitive for generating marked up content is MARKUP:

(markup 'P "Two plus three is " (markup 'strong (+ 2 3)))

which outputs the text:

<P>Two plus three is <STRONG>5</STRONG></P>

which renders as something like this in a browser:

Two plus three is 5

The generation of marked up text manages character encoding automatically, so that Unicode strings or symbols are rendered with approriate escapes in marked up documents.

The MARKUP function always produces markup with some "content" and with matching start and end tags (e.g. "<P>" and "</P>"). To produce so-called "empty tags," the functions HTMLTAG and XMLTAG can be used. They each take as arguments a sequence of "markup attributes" which are included in the output. For example:

(HTMLTAG 'IMG 'SRC "foo.png" 'ALT "This is a picture of a foo")

produces the following output:

<IMG SRC="foo.png" ALT="This is a picture of a foo">

The same call to XMLTAG produces almost exactly the same text but replaces the terminating `>' with `/>' to obey the XML specification for empty elements.

Generating markup attributes

Tags in both HTML and XML can have associated attributes which provide important additional parameters. Attributes can be specified for MARKUP by giving a list as an argument, rather than a symbol:

(markup '(P ALIGN RIGHT) "This paragraph should be flush right")

where the first element of the list is the markup tag used and the remaining elements are taken as alternating attribute names and values, so that we get this marked up output:

<P ALIGN=RIGHT>This paragraph should be flush right</P>

which would be rendered thus:

This paragraph should be flush right

By using backquote, the results of evaluation can be included in the attributes of generated markup, as in:

(markup `(FONT SIZE ,(+ 7 (get-importance)))
        "This paragraph should be flush right")

HTML builtins

In addition to these general ways of generating markup, the HTMLGEN module contains a plethora of specialized expressions for generating HTML-specific tags. The general form of these functions takes the name of the HTML markup tag as a procedure name, so that:

(H2 "This is a level " 2 " heading")

yields the marked up text:

<H2>This is a level 2 heading</H2>

Attributes can be specified by using an expression where the function name has an asterisk (*) suffix:

(OL* (STYLE "a")
  (LI "First letter")
  (LI "Second letter")
  (LI "Third letter"))

interpeting the first argument as a list of alternating attributes and values, yielding the markup:

<OL STYLE="a">
<LI>First letter</li>
<LI>Second letter</li>
<LI>Third letter</li>
</OL>

These naming conventions are changed when a given tag doesn't "make sense" without any attributes. In these cases, the unmodified form of the tag name is defined as a function which takes markup attributes, e.g. as with the anchor tag A:

(A (HREF "http://www.framerd.org/") "Visit the FramerD web site")

which yields the markup:

<A HREF="http://www.framerd.org/">Visit the FramerD web site</A>

or the FONT tag,

(P "This word should be almost " 
   (FONT (COLOR "purple") "violet"))

These HTML tags have corresponding generators in HTMLGEN:

H1, H2, H3, H4, CENTER, P, BLOCKQUOTE, OL, UL, LI, TABLE, TR, TH, TD, EM, STRONG, DEFN, TT, SUB, SUP, FONT, SPAN, DIV.

The last three generators always take an attribute argument, as described above. All of the functions, including the last three, have tag* variants which take arguments.

The following functions generate the corresponding `empty' tags:

HR, BR, IMG

interpreting their arguments as paramaters directly, e.g.

(IMG SRC "image1.png" ALT (string-append "Imagine seeing " "image.png"))

would yield the markup text:

<IMG SRC="image1.png" ALT="Imagine seeing image.png">

Generating Links

Possibly the most defining feature of the World Wide Web is the capacity for hyperlinking between documents, letting a fragment of text in one document lead directly to another document (or a fragment therein). By selecting the link in one document, a reader can have instant access to another document or fragment. In HTML, this linking is accomplished with the A element; a document generator can use a MARKUP expression to generate a link in this way:

(markup '(A HREF "http://www.whitehouse.gov/")
        "Where President " current-president " lives.")

The ANCHOR function is a special markup function intended to simplify certain patterns of hyperlinking. Its first argument is a destination which may interpreted in various ways:

• a string is simply used as a regular URL reference
• a symbol is taken as an internal reference to the current document (in HTML, this becomes a "#name" link)
• a list must be a pair of a string and a symbol, indicating a document and an internal pointer in that document

For example, the following are possible ANCHOR expressions:

(anchor "http://www.framerd.org/" "This is the FramerD web site")
(anchor '("http://www.framerd.org/docs/www-guide.html" ANCHOR)
        "This points to the description of the " 'ANCHOR " expression")
(anchor 'INTRO
        "This points to the section of the current document which is tagged as INTRO")

Passing a symbol to ANCHOR usually implies a call to the TAGGED expression elsewhere in the document. The TAGGED expression marks its content with a named tag which ANCHOR can refer to. For example,

(tagged 'intro "This is the introduction as generated on " (datestring))

The first argument to ANCHOR is evaluated, so it can be another expression whose value is the actual destination. For example, this definition produces an email address link for a particular individual.

(define (email-addr real-name email)
  (ANCHOR (string-append "mailto:" email)
          "Send electronic mail to " real-name))

and could easily be included in another document thus

(email-addr "Ken Haase" "haase@media.mit.edu")

which would generate the markup:

<A HREF="mailto:haase@media.mit.edu">Send electronic mail to Ken Haase</A>

Generating Complex Anchors

The destination of an HTML anchor does not have to be a static document existing on some remote machine. Commonly, it can refer to a virtual document generated by a program on some remote computer based on some inputs. The HTMLGEN library provides a special function for generating such references: SCRIPTURL, whose first argument is a remote URL to be invoked as a script. SCRIPTURL has two forms. The first takes one additional argument, which is a query string to pass to the script, e.g.

[fdscript] (index-url "http://www.altavista.com/search.cgi" "Scheme Hackers")
"http://www.altavista.com/search.cgi?Scheme+Hackers"

automatically encoding the string into characters valid for a URL.

The second form of SCRIPTURL is intended for use with scripts which expect a number of named inputs (rather than a single query string). These are typically scripts intended to process the input from HTML forms. As before, the first argument in this form is a string indicating the script's URL; the remaining arguments are alternating symbols (indicating named inputs) and values (indicating values assigned to them). For example:

(ANCHOR (SCRIPTURL "http://www.beingmeta.com/brico/search.fdcgi"
                    LANGUAGE "NL" WORDS "hoofd")
        "This looks up the Dutch word " "`hoofd'"
	" in the BRICO knowledge base")

generates an anchor which will list a set of frames from the BRICO knowledge base, dynamically generated by the enter.fdcgi script.

Dynamic Pages from FDScript

We just saw how to make links which point to virtual pages generated by remote scripts. The protocol for generating such remote pages is the CGI protocol which is standard on nearly all web servers. This section describes how to write such scripts using FDScript. In general, a script for generating a virtual page is an FDScript file placed on a web server, marked as executable, and starting with a line indicating where the fdcgi executable can be found, e.g. something like:

#!/usr/local/bin/fdcgi

For example when your web server is properly configured, you should be able to create a script file test.fdcgi:

#!/usr/local/bin/fdcgi
;; This is the file test.fdcgi
(define (main)
  (httpdoc
    (title "This is demonstration FDCGI script")
    (H1 "This is a heading")
    (P "This is the file " arg0 " being run at "
       (iso-timestring (xtimestamp)))))

and set it to be executable. You should then be able to access the URL http://your host/some path/test.fdcgi where some path is dependent on how the web server is configured to map URLs into local filenames. Accessing this URL will produce a page dynamically generated by the main procedure defined above. When you access the page again (or reload it from your browser), it will appear with a different date, since it is regenerated for each access.

Normally, the web server will start a new fdcgi script for each page access; however, if your server is configured to support FastCGI for FramerD scripts, fdcgi will only be run once for each separate script, processing multiple requests within the same process. This can provide much faster response and less use of system resources for starting the script over and over again.

Script Variables

Scripts generating dynamic pages can typically use input variables from a variety of sources:

• server variables are defined by the web server and include (for instance), the network address of the browser making the request
• client variables are defined by the browser making the request; these can include identity information or cookies maintaining client-side state
• script variables are defined by the script itself or the interpreter running the script; these include all of the standard fdscript functions as well as the contents of the text, xmlgen, and htmlgen modules
• request variables are defined for a particular script and generally come from the client-side "form" by which the script has been invoked

These variables are all gathered together into a single slotmap bound to CGI-DATA when the script's main procedure is executed. We could use this, for instance, to modify test.fdcgi above to report the clients address:

#!/usr/local/bin/fdcgi
;; This is the file test.fdcgi
(define (main)
  (httpdoc
    (title "This is demonstration FDCGI script")
    (H1 "This is a heading")
    (P "This is the file " arg0 " being run at "
       (iso-timestring (xtimestamp)))
    (P "You are connecting from the Internet address " 
       (get cgi-data 'remote-host))))

The following server and client variables are typically available to CGI scripts.

A good text on CGI scripting will describe these variables (which are conventions in the CGI protocol) in more detail. The only substantial difference is that those texts and their programs may use underscores (_) rather than dashes (-) in variable names.

Using Forms

To see how this works with interpreting the input from an HTML form, consider the following HTML fragment:

<H1>A silly form</H1>
<P><form action="silly-form.fdcgi">
My name is <input type='text' name='informant'>.  My favorite color is
<select name='color'>
<option>Red
<option>Green
<option>Blue
<option>Black
<option>White
</select>

<P>However, I think that `white'
<input type=radio name='white' value=yes> is
<input type=radio name='white' value=no> is not
a color

<P><input type=checkbox name='black' value=yes> I think that `black'
is a color
</form></P>

The results of this form could be processed by a file silly-form.fdcgi:

#!/usr/local/bin/fdcgi
;;; This is the file silly-form.fdcgi
(define (main)
  (httpdoc (title "Answers to silly form")
    (H1 "Hello, " (get cgi-data 'informant))
    (let ((col (get cgi-data 'color))
          (is-white-a-color? (equal? (get cgi-data 'white) "yes")))
      (P "So, your favorite color is " (font (color col) col)
         " and you think that white "
         (if is-white-a-color? "is" "is not")
         " a color")
    (if (exists? (get cgi-data 'black))
        (P "And you have an opinion about whether black is a color too"))))

For more information about writing forms in HTML, consult a document such as this summary.

The HTMLGEN module also includes primitives for generating forms. For example, a form like that above could be generated by:

(define colors '("red" "green" "blue" "black" "white"))
(define (main)
  (httpdoc (title "A silly form")
    (H1 "A silly form")
    (FORM (ACTION "silly-form.fdcgi")
      (P "My name is " (textfield 'informant) ".  "
         "My favorite color is "
         (selection (dolist (clr colors)) (option 'color clr) clr) ".")
      (P "I think that white "
         (radiobutton 'white "yes") " is"
         (radiobutton 'white "no") " is not"
         " a color")
      (P (checkbox 'black "yes") "I think that black is a color."))))

CGI-VAR and CGI-INIT

It is often more convenient way to access CGI parameters with the special forms CGI-INIT and CGI-VAR. These forms create local bindings (just like Scheme's internal DEFINEs) into which the parameter values are stored. For example, we could reimplement silly-form.fdcgi using CGI-INIT:

#!/usr/local/bin/fdcgi
;;; This is the file silly-form.fdcgi
(define (main)
  ;; This locally binds these variables to the string passed as part of
  ;; the current CGI request
  (cgi-init informant color white black)
  (httpdoc (title "Answers to silly form")
    (H1 "Hello, " informant)
    (P "So, your favorite color is " (font (color col) col)
       " and you think that white "
       (if (equal? white "Yes") "is" "is not") " a color")
    (if (not (null? black))
        (P "And you have an opinion about whether black is a color too"))))

The difference between CGI-INIT and CGI-VAR is how much the process their arguments. CGI-INIT leaves all of its arguments as strings and uses a list of strings when the client provides multiple values for a variable. CGI-VAR, on the other hand, calls a special version of the FDScript's LISP parser on the string it gets from the client; it also collects multiple values into an unordered choice rather than a list. To see how this works, consider the following simple "calculator" form:

<H1>Adding Numbers</H1>
<form action="calculate.fdcgi">
What is <input type='text' name='num1'> 
<input type='checkbox' name='op' value='+'> + 
<input type='checkbox' name='op' value='+'> - 
<input type='checkbox' name='op' value='+'> * 
<input type='checkbox' name='op' value='+'> / 
<input type='text' name='num2'>
<input type='submit' value='?'>
</form>

This could be processed by the following .fdcgi file:

(define (main)
  (cgi-var num1 num2 op)
  (httpdoc (title "Combining two numbers")
    ;; Always show the sum
    (P num1 "+" num2 "=" (+ num1 num2))
    (do-choices (each op)
      (P num1 op num2 "=" ((eval op) num1 num2)))))

Using CGI-INIT instead of CGI-VAR would have bound num1 and num2 to strings, and op to a list of strings; this would have signalled an error when + was called to add them; had it had the chance, it also would have signalled an error when it attempted to evaluate the list of strings to get an operator.

Cookies

The HTTP 1.1 protocol provides a way to maintain client-side state using cookies. A cookie is a string, stored with the client's browser, which is sent along with requests to identify the browser and client. Cookies may be set automatically (though sometimes, browsers ask the user to confirm the acceptance of a cookie) when certain documents are read and then sent along automatically with subsequent requests.

The function (SET-COOKIE! symbol value [domain+path] [expires]) includes a cookie with the current page being generated. If the page is read on a browser, subsequent requests from the browser will bind the CGI variable symbol to value. The additional arguments limit the scope of the cookie so that it is only sent back to a particular path or to particular domains or up till a specified time. The argument domain+path can have the form of a qualified domain name (in which case all hosts in the specified domain will receive the cookie), a slash-separated pathname (in which case scripts beneath that path on the current server will receive the cookie), or both together (in which case scripts beneath the path on any hosts in the domain will receive the cookie).

If the value value is a string, it is used directly (with escapes in and out of Unicode handled automatically); otherwise, it is converted to a string and set as a string when subsequently received. The function parse-arg can be used to convert it back to a Lisp object.

Frames to Anchors

As we described above, one of the greatest things about the World Wide Web is the ability to link between documents. Comparably, one of the greatest things about FramerD and FDScript is the easy ability for objects (frames) to refer to other frames. These two aspects are connected whenever HTML generation needs to display a frame reference (an OID) on a web page. The frame reference is "made live" by converting it into an anchor which refers to a browse script directed at the OID. The browse script is just an FDCGI script (actually it could be any kind of script, but practically it is nearly always an FDCGI script) which interprets its argument as a frame reference and describes the corresponding frame. When everything comes together, a user can just click on a frame description and --- voila! --- get a description of the frame.

The default browse script used in generating such "live links" is called browse.fdcgi, which browsers normally interpret with respect to the path of the URL under which the reference appears. This means that if a script in one directory displays a reference to a frame, it will refer to the script browse.fdcgi in the current directory.

The procedure USE-BROWSE-SCRIPT! sets the default browse script or to set a special browse script for a particular OID or a particular pool. Different browse scripts may be customized to different applications, highlighting different aspects of the frame or even presenting just the information in the frame without any indication of its "framishness". For example, the GNOSYS knowledge management application uses a browse script which renders frames describing web pages into a reconstructed version of the web page, subtly annotated with pointers into GNOSYS' knowledge base.

When provided, the second argument to USE-BROWSE-SCRIPT! can be an OID or a pool, specifying that the first argument (a URL) be used to browse either the particular OID or all OIDs in the designated pool.

In any case, the basic framework for a browse script is very simple. For example, the following browse script simply displays a table of slot values.

#!/usr/local/bin/fdcgi
;; This is the file sbrowse.fdcgi
(use-browse-script! "sbrowse.fdcgi")
(define (main)
  (cgi-var QUERY) ;; Get the query string
  (httpdoc (header 'pragma "no-cache") ; the browser shouldn't save this
    (TITLE "Description of the frame " QUERY)
    (H1 "Description of the frame " QUERY)
    (table
      (doslots (f slot value QUERY)
        (tr (th slot) (td value)))))))

This kind of tabular display is also generated by the special HTML generation procedure FRAME->HTML; the script above could have been rewritten simply as:

#!/usr/local/bin/fdcgi
;; This is the file sbrowse.fdcgi
(use-browse-script! "sbrowse.fdcgi")
(define (main)
  (cgi-var QUERY) ;; Get the query string
  (httpdoc (header 'pragma "no-cache") ; the browser shouldn't save this
    (TITLE "Description of the frame " QUERY)
    (H1 "Description of the frame " QUERY)
    (FRAME->HTML QUERY)))

In fact, the display would probably look better than the simple tabular format produced by the original version since FRAME->HTML takes advantage of a few HTML bells and whistles).

The default generation of live links from frames can always be overridden by an explicit ANCHOR expression combined with a URL generated by SCRIPTURL, e.g.

(anchor (scripturl "alternate-browse-script.fdcgi" frame)
        (get frame 'obj-name))

Also, while OIDs are normally linked to browse scripts, generated documents sometimes have descriptions of several OIDs within the document and it would be better to have clicking on the OID go to another location in the currently generated document. When the argument to a TAGGED expression [ref] is a frame, it creates just such an internal link target; subsequent presentation of the frame will turn into a link to the internal pointer rather than external script invocation. To make things easier still, descriptions of frames generated by FRAME->HTML generates an automatic tag for internal reference.

The only problem with this is that the description itself (from TAGGED or FRAME->HTML) needs to be generated before any references to the OID it describes (so the reference can know that there is an internal pointer). To get around this particular problem, the procedure (DECLARE-LOCAL-FRAME! frame) asserts that frame will be an internal reference, forcing the generation of OID anchors to be internal.

Details of HTTP generation

When generating a document, FDScript divides document generation into four distinct phases:

1. HTTP header generation; this is where cookies are set and URL redirection can happen
2. HTML HEAD generation; this is where titles, metadata, and stylesheets (among other properties) are emitted
3. HTML BODY generation; this is where what the user sees is generated
4. HTML post generation; this is normally empty, but is kept to signal errors when attempts to output content occur after the document is finished

HTTP Header Phase Procedures

The function HEADER outputs an HTTP header and signals an error if it is too late to do so. One particularly useful application is to provide a pragma header of "no-cache" to keep the browser from caching the contents of a dynamically changing page. It would be emitted by this expression:

   (header 'pragma "no-cache")

The first argument is a symbol or string naming the header to be sent; the second argument is a string containing the actual content. Note that header (like the other header phase procedures) must be called before any HTML primitives (either content generating primitives like P or metadata primitives like TITLE) are called.

The function (HTTP-REDIRECT url) generates a header which redirects the browser to url.

The function (HTTP-SPLASH url interval) emits a Refresh: header which will cause the page to reload from url every interval seconds.

HTML HEAD Phase Procedures

The function (TITLE ...args) adds a title element to the header; this is often displayed at the top of the browser frame. The arguments are processed PRINTOUT-style:

  (title "This document was generated under "
         (get (resources) 'os) " on " (get-day))

would produce markup of the form:

<TITLE>This document was generated under i686-unknown-linux on SATURDAY</TITLE>

The function (STYLESHEET! local or global url reference) adds a declaration to the header indicating which stylesheet to use for displaying this document. For example

  (stylesheet! "fdsite.css")

The function (META var val) specifies a META HTML header (such as KEYWORDS or author). For example:

  (META 'keywords "knowledge,poetry,natural language")

The function (SET-DOCTYPE! type location) configures the !DOCTYPE header used in a generated document. The type argument should be a symbol while the location argument should be either a URL or a standard reference to be recognized by the browser (starting with "+" or "-").

Configuring your Web Server

Disclaimer: This section is a provisional guide to setting up an Apache web server to use fdcgi scripts. It explains what I've done for my web servers, but it may not be the best way to do things. I'm not an expert Apache hacker.

To set up your web server to run fdcgi scripts in a particular directory, you need to tell Apache two things:

1. it's okay to execute scripts in the directory where your .fdcgi files live
2. files ending in .fdcgi should be handled as CGI scripts

To do the first, add the following to your web server's httpd.conf configuration file:

<Location /dir>
Options ExecCGI
AddHandler cgi-script .fdcgi
</Location>

where dir is the directory where your files live.

If you want to be able to have .fdcgi scripts anywhere, you can add the following to the access.conf file:

<Location />
Options ExecCGI
AddHandler cgi-script .fdcgi
</Location>

Using mod_fdserv

FramerD can be used, with the Apache mod_fdserv module, to implement persistent multi-threaded web applications. This module can be installed through either a bundled RPM or DEB package or by typing make fdserv-install in the root of the FramerD build tree. It can also be downloaded directly in source form and compiled and installed with apxs>.

    AddHandler fdservlet .fdcgi .fdxml

AddHandler fastcgi-script .fdcgi

Options ExecCGI
AddHandler fdcgi .fdcgi
AddHandler fdcgi .fdxml