Permalink

README for FossMD

Test suite for FossMD

Quick start

Just run test.sh without any arguments to run all tests.

Summary

This test suite generates HTML and text files using FossMD, and then compares the generated output with the expected output.
The comparision between generated and expected output is "dumb" in the sense that a simple diff -q should suffice to detect success or error.
Any necessary text filtering is done when creating the output, not when comparing it.

A separate test suite validates generated HTML, or directly against a (Fossil) web server.

Purpose

The main purpose for this trest suite is to check the Markdown-to-HTML conversion.
These test use source documents as input.
This includes standard Markdown documents, Wiki documents, and documents with Markdown extensions that generate SVG output from JavaScript.

The test suite also checks the user interface, both the CLI and the web interface.
The CLI test launches existing subcommands, but without the necessary arguments, to check that the correct help message are shown for each command.
It also check for the correct error message for non-existing subcommands.
The user interface test for web pages is simple, as it only checks for existing and non-existing web pages.
The CLI and web interfaces do not require any source documents as input.

NOTE:
Not all commands or webpages are included in this test. For example the zip command and the /zip webpage creates an output that is too big to handle, binary and thus hard/impossible/useless to compare, and will most probably change from one test run to another.

The validation part only checks the HTML syntax (using a W3C HTML validator) not the contents.

Details

The test suites are separated into these components:

1. Get and save the output: test-get.sh

   This script just calls the other test-get-* scripts to generate output files based on input files.

   a. test-get-md.pl : Get standard Markdown input (test-in/md) and save HTML output (test-out/md).
   b. test-get-wiki.pl : Get Wiki input (test-in/wiki) and save HTML output (test-out/wiki).
   c. test-get-js.py : Get extended Markdown/HTML input (test-in/js) and save client-side (browser JavaScript) generated HTML and graphic SVG content (test-out/js). Uses Selenium and XFvb.
   d. test-get-cli.pl : Launch subcommands from the CLI and save the (help or error message) output (test-out/cli).
   e. test-get-www.pl : Access the web interface by HTTP and save two documents per request (both documents saved in test-out/www):

   1. The first line in the HTTP (the HTTP Status Code)
   2. The entire HTML, excluding the HTTP header
      

2. Compare the output: test-diff.sh

   Compare all saved output with the expected output.

3. Validate the HTML files: test-validate.sh

   Check that all files are valid HTML, using the The Nu Html Checker.

1. Get generated HTML : test-get.sh

Usage:

test-get.sh [--options=...]

Help:

test-get.sh -h|--help

Requirements:

See requirements for each test-get-* script below.

Details:

This script is just a wrapper for running all test-get-* scripts, described below.

1a. Get standard Markdown-to-HTML generated output : test-get-md.pl

1b. Get Wiki-to-HTML generated output : test-get-wiki.pl

Usage:

test-get-md.pl [--options=...]
test-get-wiki.pl [--options=...]

Help:

test-get-md.pl -h|--help
test-get-wiki.pl -h|--help

Requirements:

These two test suites are listed together, as they are very similar.
They are easy to setup, as few requirements are needed:

• The fossil fossil-fossmd, and/or fossmd binaries, depending on which one(s) should be tested.
• A fossil repository, containing the input Markdown/Wiki files.

Note that the fossil/fossil-fossmd/fossmd web server doesn't have to be running.

Details:

All tests cases in test-get-md.pl and test-get-wiki.pl use the command fossil http to obtain the HTML:

    printf "%s\n" "GET $myurl HTTP/1.0" | fossil http

As this is a HTTP command, the output includes both the HTTP header and the body (the web page).
We are only interested in the HTML output generated from Markdown and Wiki pages.

An example Markdown and Wiki page, respectively, parsed by fossil (and fossil-fossmd/fossmd should behave the same):

Markdown:

    HTTP HEADER, HTML HEADER, AND SOME HTML BODY ELEMENTS
    <div class="content">
        <script>
            function gebi(x){
            if(x.substr(0,1)=='#') x = x.substr(1);
            var e = document.getElementById(x);
            if(!e) throw new Error('Expecting element with ID '+x);
            else return e;}
        </script>
        <div class="markdown">
            <p>Markdown Formatting Rules</p>
            <p><a href="http://fossil-scm.org/index.html/md_rules">http://fossil-scm.org/index.html/md_rules</a></p>
        </div>
    </div>
    HTML BODY FOOTER

Wiki:

    HTTP HEADER, HTML HEADER, AND SOME HTML BODY ELEMENTS
    <div class="content">
        <script>
            function gebi(x){
            if(x.substr(0,1)=='#') x = x.substr(1);
            var e = document.getElementById(x);
            if(!e) throw new Error('Expecting element with ID '+x);
            else return e;}
        </script>
        <p>Wiki Formatting Rules
        <p><a href="http://fossil-scm.org/index.html/wiki_rules">http://fossil-scm.org/index.html/wiki_rules</a>
    </div>
    HTML BODY FOOTER

To keep it simple, both the parsing and comparision of the HTML output, strip the HTML from anything outside the <div class="content"> tag.
Also strip any present <script> tag inside the <div class="content"> tag. That gives us a reduced output, easier to read:

Markdown:

    <div class="content">
        <div class="markdown">
            HTML PARSED FROM MARKDOWN
        </div>
    </div>

Wiki:

    <div class="content">
        HTML PARSED FROM WIKI
    </div>

This means that all files in the test-out-expected directory start with the line <div class="content">. In the case of Markdown, the second line will always be <div class="markdown">.

Wiki format restriction

As the Fossil Wiki Formatting Rules indicates, Wiki content is more restricted than Markdown.
Especially, any attributes that specify javascript or CSS are elided.
This restriction should be tested, so that HTML content generated from a Wiki page never would contain any CSS nor JavaScript tags.

1c. Get client-side (JavaScript) generated HTML : test-get-js.py

Usage:

test-get-js.py [--options=...]

Help:

test-get-js.py -h|--help

Requirements:

This test suite may be more tricky to setup, as it needs many requirements:

• A local fossil/fossil-fossmd/fossmd server running on localhost:8080, serving the repository with the Markdown/Wiki sources
• Firefox (to parse the client-side JavaScript)
• geckodriver (for now, we will only run tests using one single browser, Firefox)
• Selenium (Python flavour)
• Xvfb (optional but convenient, to run headlessly; not available on MS Windows)
• Optional: The pgrep and pkill tools (installed by default on any Linux/UNIX flavour, but not on MS Windows/MinGW/MSYS2; available from the procps MSYS2 package)
  These tools are required to conditionally auto-start-and-stop the web server in case it hasn't been started before running this script.
  If the web server always is running when executing this script, pgrep and pkill are not required.

Besides testing correct parsing of extended Markdown to HTML and JavaScript-generated SVG output,
this suite should also test that Wiki documents do not generate the SVG, due to the Wiki format restricions for JavaScript.

Some of the input test files are pure HTML. They should generate SVG output regardsless of which web server is running.
Other input files are Markdown containing Fenced code blocks.
which to indicate that

When the running server is fossil, Besides testing correct parsing of extended Markdown to HTML and JavaScript-generated SVG output,

1d. Get output from CLI commands : test-get-cli.pl

Details for test-get-cli.pl

Usage:

test-get-cli.pl [--options=...]

Help:

test-get-cli.pl -h|--help

Requirements:

• The fossil, fossil-fossmd, and fossmd binaries

• Perl module Regexp::Common
  Install (CPAN shell):

  > cpan
  install Regexp::Common
  

  or (Ubuntu):

  sudo apt-get install libregexp-common-perl
  

Details

This test suite gets output from existing and non-existing fossil-fossmd and fossmd CLI subcommands.
No input files from test-in are used.
Instead, the "input" for this test suite is a CLI command.
The output is either a help message (existing commands), or a error message (non-existing commands).

To get a list of all existing CLI commands, there are basically two ways:

1. Parse the output from fossil-fossmd help -a

2. Parse the source code for comments formatted as commands, as described in Adding Features To Fossil

   /*
   ** COMMAND: xyzzy
   **
   ** Help text goes here.
   */
   

All the gory details about how Fossil internally parses commands and webpages are done in src/mkindex.c.
When parsing files for commands/webpages, exclude src/mkindex.c.

The two ways of getting the command list should return identical results.
One of the tests in this suite is to check that the lists are identical.

This script should test both the CLI for fossil-fossmd and fossmd, as the list of available commands differs between fossil-fossmd and fossmd.
Both existing and non-existing commands should be tested.

1e. Get output from web pages : test-get-www.pl

Usage:

test-get-www.pl [--options=...]

Help:

test-get-www.pl -h|--help

Requirements:

• The fossil-fossmd binary
• A fossil repository, containing any files.
• The fossil-fossmd web server doesn't have to be running.

Details

This test suite gets web pages from both fossil-fossmd and fossmd.
The method obtaining the output for this test suite is very similar to test-get-md.pl and test-get-wiki.pl.
The difference is that no input files from test-in are used.
Instead, the "input" for this test suite is a URL accessing the local fossil-fossmd or fossmd web server.
Output is: standard web pages, user-created documents, and non-existing webpages.

The output is the entire HTTP Response, both the HTTP header and the body (the web page).
Anyway, some text is removed, as for example the Date: HTTP Header field (which always differs), and the footer version info (which may differ).

The first line should be a HTTP Status Code, either 200 OK or 404 Not Found.
The exceptions are the following pages, which should return 302 Moved Temporarily:

- /
- /home
- /index
- /not_found

Other exceptions:

- /zip
- /json??? %%%

To get a list of all existing web pages, there are basically two ways:

1. Parse the output from fossil-fossmd help -w

2. Parse the source code for comments formatted as web pages, as described in Adding Features To Fossil

   /*
   ** WEBPAGE: helloworld
   */
   void helloworld_page(void){
     style_header("Hello World!");
     @ <p>Hello, World!</p>
     style_footer();
   }
   

The two ways of getting the web page list should return identical results.
One of the tests in this suite is to check that the lists are identical.

To access a standard GUI web page, add the prefix /:

    url=timeline
    printf "%s\n" "GET /$url HTTP/1.0" | fossil-fossmd http

To access a user-created file in the repository as a web pages, add the prefix /doc/ckout/ to the file:

    url=README.fossmd.md
    printf "%s\n" "GET /doc/ckout/$url HTTP/1.0" | fossil-fossmd http

This script should test both the web page list for fossil-fossmd and fossmd, as the list of available web pages differs between fossil-fossmd and fossmd.
Both existing, user-created, and non-existing web pages should be tested.

2. Compare output : test-diff.sh

This script compares the generated output (test-out) with the expected output (test-out-expected).

Usage:

test-diff.sh [--options=...]

Help:

test-diff.sh -h|--help

Requirements:

• The diff tool (installed by default on any Linux/UNIX flavour, but not on MS Windows/MinGW/MSYS2)

Details

By design, this script is very simple.
It only prints a resume of files that differ, if any.
To get a better overview over diffing files, it is recommended to use an external graphical diff tool able to compare directories, such as Meld, KDiff3, WinMerge, vimdiff etc.

For example, using Meld:

    meld test-out test-out-expected &

or, to include the input directory as a reference:

    meld test-in test-out test-out-expected &

A quick check to find pages which got the wrong output:

    grep 200 test-out/www/* | cut -d : -f1|grep -v 200 | xargs grep '200 OK'
    grep 302 test-out/www/* | cut -d : -f1|grep -v 302 | xargs grep '302 Temporarily Moved'
    grep 403 test-out/www/* | cut -d : -f1|grep -v 403 | xargs grep '403 Forbidden'
    grep 404 test-out/www/* | cut -d : -f1|grep -v 404 | xargs grep '404 Not Found'

3. Validate HTML files: test-validate.sh

This script validates HTML files served by a local web server, fossil, fossil-fossmd orfossmd.
Files are validated using the The Nu Html Checker.
This means that this test does not use any input/output files in test-in, test-out, or test-out-expected.

Read more: The Nu Html Checker README

Usage:

test-validate.sh [--options=...]

Help:

test-validate.sh -h|--help

Requirements:

• Java 8
• The Nu Html Checker
  
• Optional: The pgrep and pkill tools (installed by default on any Linux/UNIX flavour, but not on MS Windows/MinGW/MSYS2; available from the procps MSYS2 package)
  These tools are required to conditionally auto-start-and-stop the web server in case it hasn't been started before running this script.
  If the web server always is running when executing this script, pgrep and pkill are not required.

Details

The Nu Html Checker may be used either has a validator for HTML files, or as a HTTP client for a server.

In the second case, a local fossil-fossmd server must be running on localhost:8080, and the same URL list as used in test-get-www.pl may be used to validate the HTML.

The first method does need any server running, and is probably faster, but the HTML files needs to be saved locally.
The HTML files to validate are obtained from test-get-www.pl (fast, using fossil-fossmd http) and saved in test-out/www.

The second method (which we use) requires a local server to be running, which probably is a slower way to obtain the HTML.
The advantage with a running server is that the test suite may be extended with other W3C tools, such as the Link Checker and the CSS Validator.

List of included tests

Markdown (standard)

Included tests by test-get-md.pl:

By default, all files in test-in/md/

DocBook features:

• Table-of-Contents (ToC)
• Single/Split pages
• Navigation bars (navbars) at top and bottom of pages (when page in split)
• Page number

Markdown features currently not supported by Fossil:

• Enhanced Markdown (MultiMarkdown) to HTML conversion:
  • Fenced code blocks
  • File Transclusion
• Tables:
  • Table with caption and grouping
• Abbreviations

Markdown features already supported by Fossil:

To test features already supported by Fossil, the corresponding previously saved HTML page can be generated by Fossil, and then filtered.

• Standard Markdown (John Gruber's) to HTML conversion (already supported by Fossil)
• Tables:
  • "Simple" table
  • Table with added leading and tailing pipes to each line
  • Table with mixed alignment
  • Table with formatted cells

Wiki

Included tests by test-get-wiki.pl:

TODO

Included tests: Markdown (extensions, using JavaScript)

Included tests by test-get-js.py:

TODO

Included tests: CLI

Included tests by test-get-cli.pl:

TODO

Included tests: www

Included tests by test-get-www.pl:

TODO

Related: Fossil test suite

To test all Fossil features, compile with all features enabled:

cd /path/to/fossil
./configure --debug --with-openssl=auto --with-zlib=auto --with-th1-docs --with-th1-hooks --with-tcl=1 --with-tcl-private-stubs=1 --fossil-debug --json
make
make test

All tests above should also succeed for fossil-fossmd. Anyway, fossmd is a heavily stripped-down version of fossil-fossmd, and should therefore run a reduced test suite.

TODO - use Transclusion: Prerequisites and setup for client-side generated HTML

This file should include other Markdown files in this directory, like this:

Transclusion example for parsed text:

{{README.geckodriver.md}} {{README.selenium.md}} {{README.Xvfb.md}}

Transclusion example for non-parsed text:

{{relative/path/to/some_other_file.txt}}