Pantangle

This document describes a program called, pantangle. Pantangle is a literate programming tool to tangle source code from pandoc markdown documents.

When literate programming was first developed by Donald Knuth, the process of producing the output involved two processing steps. A tangle processing step produced the code file given to the compiler. A weave step produced the printable document. Document production was focused, naturally enough, on using the TeX documentation system.

Other alternatives have evolved since 1984. With the advent of markup languages that have specific syntax to denote program source code, the weaving step can be eliminated. The literate program source is just the document file. It is, of course, necessary to tangle out the source code. The tangle processing is significant in that it separates the order of the document from any ordering requirements of the language processor (compiler or interpreter).

The pantangle program is just such a tangling tool. The literate program source is just a pandoc source file. Within the pandoc source, the markup for code sections are used as the syntactic mechanism to obtain the program source code from literate program document file.

Chunk Syntax

Pantangle was developed and tested using the pandoc variation of markdown. Since pandoc accepts many formats of input file, it is possible to use other input file formats as long as the parsing yields AST blocks of type CodeBlock.

Pandoc also support several variations on code block syntax. Here we use the fenced code blocks extension with backticks.

A literate program chunk can be defined by a code block that has an identifier. For example:

In this example, my-chunk-name is the identifier of the code block and is used as the name of the literate program chunk.

A literate program chunk can be referenced by enclosing the chunk name in << and >> and placing the reference on a separate line in the code text of another chunk. For example:

During the tangling processing, the my-command.tcl chunk can be specified as a tangle root and pantangle produces a file named, my-command.tcl, where <<my-chunk-name>> has been substituted with the code text defined previously, i.e. the file my-command.tcl would contain:

proc addOne {value} {
    return [incr value]
}
puts [addOne 15]

The remainder of this document contains the literate program for pantangle and serves as a more complex example.

Operational Details

The operation of pantangle is based on the ability of pandoc to parse input files and emit a serialization of the AST (Abstract Syntax Tree) that is the result of the parsing. Pantangle can be requested to encode the AST in JSON. The JSON is then converted into a Tcl dictionary using the json package in the Tcl standard library. The only nodes in the AST that are of interest are those of type, CodeBlock.

A CodeBlock node can have a number of properties along with the code text itself. The only property used by pantangle is the identifier of the block. This identifier is used as the name of the literate programming chunk contained in the CodeBlock. If multiple literate chunks have the same name, they are concatenated together in the order in which they are encountered in the AST. Any CodeBlock whose identifier is empty, is ignored by pantangle.

After collecting all the CodeBlocks from the pandoc AST, each tangle root is composed by looking for literate chunk references in the text of a given root chunk. References must be on a line by themselves, preceded or followed by optional whitespace. Whitespace before the start of the chunk reference is used as the indentation of the chunk text when it is substituted for the reference. Whitespace after the end of the chunk reference is ignored. The chunk name contained in the markers may also be preceded or followed by whitespace, which is discarded.

The list of tangle roots, given as an option to pantangle, is then validated. Each tangle root is the root of an implied directed graph connected by the chunk references. This graph is built and validated to ensure that the directed graph is acyclic. A cycle in the tangle root graph would cause an infinite recursion when the references are substituted.

After validation, each tangle root is recursively substituted based on the chunk references found in the root chunk text. The result is then written to a file with the same name as the root chunk.

Command Interface

Options

The pantangle program accepts the following options to control its behavior.

Command Line Options Parsing

The Tcl standard library package, cmdline is used to parse the command line arguments.

package require cmdline

set optlist {
    {roots.arg {} {List of tangle tree roots (whitespace separated).}}
    {outdir.arg {.} {Directory name where output files are placed. Default is current working directory.}}
    {pandoc.arg {} {Additional arguments passed to pandoc.}}
    {level.arg notice {Logging level.}}
    {strict {Require all referenced chunks to be defined and all defined chunks to be referenced.}}
    {summary {Print summary of literate program chunk usage.}}
    {graph.arg {} {Output the tangle roots as a directed graph using the given format.}}
    {version {Print version and exit.}}
}

try {
    set usage "\[options] ?file1 file2 ...?\noptions:"
    array set ::options [::cmdline::getoptions argv $optlist $usage]
} trap {CMDLINE USAGE} {msg} {
    chan puts -nonewline stderr $msg
    exit 1
}

Argument Validation

We check for the version option first. If it is present, then pantangle just prints the version and exits.

if {$::options(version)} {
    chan puts "[::cmdline::getArgv0] $::version"
    exit 0
}

We insist that a -roots option be supplied. Otherwise, there are no specified tangle roots to generate.

if {[string length $::options(roots)] == 0} {
    chan puts stderr "a '-roots' option is required"
    exit 1
}

Check early on that if -outdir exists then it must be a directory file. If it doesn’t exist, then it is created.

if {[file exists $::options(outdir)]} {
    if {![file isdirectory $::options(outdir)]} {
        chan puts stderr "output directory, '$::options(outdir)' is not a directory file"
        exit 1
    }
} else {
    file mkdir $::options(outdir)
}

Logging

The Tcl standard library package, logger, and its companions are used to provide logging facilities.

package require logger
package require logger::utils
package require logger::appender

The following code sequence initializes the logging in a child namespace call, log.

set logger [::logger::initNamespace [namespace current]]
set appenderType [expr {[dict exist [fconfigure stdout] -mode] ?\
        "colorConsole" : "console"}]
logger::utils::applyAppender -appender $appenderType -serviceCmd $logger\
        -appenderArgs {-conversionPattern {\[%c\] \[%p\] '%m'}}

log::setlevel $::options(level)

Main

Once the command line interface is handled, the processing order of the operation of pantangle follow the discussion of the operational details.

proc main {} {
    openPandocChan
    buildLiterateChunks
    buildChunkGraph
    tangleCodeFromChunks
    validateTangle
    summarize
}

Pandoc AST

One of the many output formats from pandoc is json. This produces the AST of the input document in JSON format. We open a pipeline to a process running pandoc and read its standard output to obtain the JSON encoded AST.

Running Pandoc

It is possible to feed additional arguments to pandoc via the -pandoc option, but they are seldom needed. When using additional arguments to pandoc do not specify an output format. That is done here. Remember, we are only using pandoc to obtain the AST for the document. Pantangle does not do any document production. To produce a printable document requires running pandoc itself on the document file.

proc openPandocChan {} {
    try {
        variable pandoc_chan
        set cmd_line [string cat "|pandoc $::options(pandoc) -t json " $::argv]
        log::debug "pandoc command: '$cmd_line'"
        set pandoc_chan [open $cmd_line]
    } on error {result opts} {
        log::error $result
        return -options $opts
    }
}

Parsing the AST

Since we requested the AST produced by pandoc to be encoded in JSON, we use the Tcl standard library json package to parse the JSON into a Tcl dictionary that contains all the information.

package require json

We don’t explain the pandoc AST in detail here. We are only interested in the blocks component in the AST and only in those blocks that are of type, CodeBlock. The contents of a CodeBlock node is a two-item list consisting of the attributes of the block and the text of the block. The attributes of the CodeBlock are another list, the first element of which is the identifier of the block. This identifier is extracted and used as the chunk identifier. The result of walking through all the blocks is to build a Tcl dictionary whose keys are the chunk identifiers and whose corresponding values are the code text of the block.

proc buildLiterateChunks {} {
    variable pandoc_chan
    variable literate_chunks [dict create]

    try {
        set ast_json [chan read $pandoc_chan]
        chan close $pandoc_chan
        set parsed [::json::json2dict $ast_json]
        # The document blocks are in the parsed dictionary keyed by "blocks".
        set blocks [dict get $parsed blocks]

        # The blocks section of the document is just a list of the
        # individual blocks in the document. Each block has a type
        # field that determines its interpretation.
        foreach block $blocks {
            dict update block t block_type c block_contents {
                if {$block_type eq "CodeBlock"} {
                    lassign $block_contents attrs code_text
                    set chunk_id [string trim [lindex $attrs 0]]
                    # If no identifier is given, then the block is ignored.
                    if {[string length $chunk_id] != 0} {
                        dict append literate_chunks $chunk_id $code_text \n
                    }
                }
            }
        }
    } on error {result opts} {
        log::error $result
        return -options $opts
    }
}

Validating Root Chunk Graphs

Each tangle root of a literate program involves references to other literate program chunks. These references create a directed graph. In large literate program files where multiple roots are tangled out and where the same text may be used in several files (e.g. a license statement), the graph starting at the tangle root could potentially have a cycle. A cycle in the graph implies an infinite loop when the tangled output code is produced.

We want to catch any potential cyclic graphs of tangle roots to prevent the infinite recursion. The Tcl language does eventually detect what it considers a infinite loop, but we can do better by building and analyzing the graph.

Again, the Tcl standard library has a graph package that does the heavy lifting.

package require struct::graph

We build the graph of each root chunk separately and each is validated separately.

proc buildChunkGraph {} {
    variable literate_chunks

    foreach root $::options(roots) {
        if {![dict exists $literate_chunks $root]} {
            set msg "unknown root chunk: '$root'"
            log::error $msg
            throw UNKNOWN_ROOT $msg
        }

        set chunk_graph [::struct::graph]
        $chunk_graph node insert $root
        graphChunkReferences $chunk_graph $root
        try {
            validateChunkGraph $chunk_graph $root
        } finally {
            dotGraph $chunk_graph $root
            $chunk_graph destroy
        }
    }
}

To build the graph of a root chunk, it is necessary to inspect the text of the chunk and find any literate chunk references.

A regular expression is sufficient to identify a chunk reference in the text of the block. Since we use the expression in multiple locations it is factored into a namespace variable.

variable chunk_regexp {^(\s*)<<(.+)>>\s*$}

proc graphChunkReferences {graph node} {
    variable literate_chunks
    variable chunk_regexp

    set code_text [dict get $literate_chunks $node]
    set code_lines [split $code_text \n]
    foreach line $code_lines {
        if {[regexp $chunk_regexp $line _ leading_space chunk_id]} {
            set chunk_id [string trim $chunk_id]
            if {[dict exists $literate_chunks $chunk_id]} {
                # Avoid duplicating nodes in the graph for those chunks that
                # have already been referenced. If a chunk is in the graph
                # it has been scanned. Here we just need another edge to connect
                # it to the current graph node.
                if {[$graph node exists $chunk_id]} {
                    $graph arc insert $node $chunk_id
                } else {
                    $graph node insert $chunk_id
                    $graph arc insert $node $chunk_id
                    graphChunkReferences $graph $chunk_id ; # Continue recursively.
                }
            }
        }
    }
}

To validate a root chunk graph, we perform a depth first search (DFS) classifying the edges of the graph. If any edge in the graph is classified as a back edge, then the graph is not acyclic and is rejected.

proc validateChunkGraph {graph root} {
    dfs $graph $root
    log::debug "$root graph:\n[$graph serialize]"
    set back_arcs [$graph arcs -key type -value back]
    log::debug "back arcs: $back_arcs"
    if {[llength $back_arcs] != 0} {
        log::error "graph for $root contains a cycle"
        foreach back_arc $back_arcs {
            lassign [$graph arc nodes $back_arc] source target
            log::error "arc from $source to $target forms a cycle"
        }

        throw NOT_DAG "$root is not an acyclic directed graph"
    }
}

This DFS procedure classifies the edges and records the pre-order and reverse post-order of the nodes during the traversal of the graph. We do not use the node orders in this application.

proc dfs {graph root} {
    set nodes [$graph nodes]
    log::debug "nodes: $nodes"
    foreach node $nodes {
        $graph node set $node pre 0
        $graph node set $node rpost 0
    }
    variable preorder 1
    variable postorder [llength $nodes]
    classifyNode $graph $root

    return
}

The heart of the DFS is the edge type classification and the node order computation. Note that it is a recursive algorithm.

proc classifyNode {graph node} {
    variable preorder
    $graph node set $node pre $preorder

    foreach arc [$graph arcs -out $node] {
        set succ [$graph arc target $arc]
        set succPre [$graph node get $succ pre]
        if {$succPre == 0} {
            $graph arc set $arc type tree
            classifyNode $graph $succ
        } elseif {[$graph node get $succ rpost] == 0} {
            $graph arc set $arc type back
        } elseif {$preorder < $succPre} {
            $graph arc set $arc type frwd
        } else {
            $graph arc set $arc type cross
        }
    }

    incr preorder
    variable postorder
    $graph node set $node rpost $postorder
    incr postorder -1

    return
}

The Tcldot package provides a command interface to the dot family of programs for graph layout. Using commands from the Tcldot package means that it is not necessary to know the details of the dot language to specify the graph. The graph is built up by invoking commands and, when complete, the graph can then be rendered into any supported graphical file format. The transformation from Tcllib graph command to Tcldot dot commands is direct since both deal directly with nodes and edges.

proc dotGraph {graph root} {
    package require Tcldot

    set graph_type $::options(graph)
    if {$graph_type eq {}} {
        return
    }

    set dot_graph [dotnew digraph]
    $dot_graph setnodeattributes shape box
    $dot_graph setnodeattributes style filled
    $dot_graph setnodeattributes fillcolor yellow

    foreach node [$graph nodes] {
        set dot_node [$dot_graph addnode $node]
    }

    foreach arc [$graph arcs] {
        lassign [$graph arc nodes $arc] source target
        set dot_edge [$dot_graph addedge $source $target]
        # This is where each edge is labeled by its type as found by the dfs procedure.
        # The intent is to help identify where any potential back edges arise.
        # The extra blank space in the edge label helps in placement of the
        # label relative to the spline that shows the edge.
        set edge_type " [string toupper [string index [$graph arc get $arc type] 0]]"
        $dot_edge setattributes label $edge_type
    }

    set outfile [file join $::options(outdir) [file rootname $root].$graph_type]
    set outchan [open $outfile w]
    try {
        $dot_graph write $outchan $graph_type
    } finally {
        chan flush $outchan
        chan close $outchan
        $dot_graph delete
    }

    return
}

Assembling the Code

Finally, we get to the part of pantangle that extracts the code from the pandoc document and generates the file to hand to the language processor. The design is a recursive walk starting at the root chunk scanning for chunk references. This is similar to what was done previously when building the directed graphs of the root chunks.

While traversing the chunks, we record the number of times a chunk is referenced and whether we find references to undefined chunks. These values are used later to validate whether the literate source code file is consistent.

proc tangleCodeFromChunks {} {
    variable literate_chunks
    variable undefined_references [dict create]
    variable chunk_references [dict create]

    foreach chunk_id [dict keys $literate_chunks] {
        dict set chunk_references $chunk_id 0
    }

    foreach root $::options(roots) {
        set result {}
        if {![dict exists $literate_chunks $root]} {
            log::error "unknown root chunk: '$root'"
            dict incr undefined_references $root
            continue
        }
        dict incr chunk_references $root

        set outfile [file join $::options(outdir) $root]
        set outchan [open $outfile w]
        try {
            set code_text [dict get $literate_chunks $root]
            puts $outchan [expandChunk $code_text {}]
        } finally {
            chan close $outchan
        }
    }
    return
}

The code to scan the code text in a chunk for chunk references is similar to that used in the graphChunkReferences procedure.

proc expandChunk {code_text leader} {
    variable literate_chunks
    variable undefined_references
    variable chunk_references
    variable chunk_regexp

    set code_lines [split $code_text \n]
    foreach line $code_lines {
        if {[regexp $chunk_regexp $line _ leading_space chunk_id]} {
            set chunk_id [string trim $chunk_id]
            if {[dict exists $literate_chunks $chunk_id]} {
                dict incr chunk_references $chunk_id
                set chunk_text [dict get $literate_chunks $chunk_id]
                append result [expandChunk $chunk_text [string cat $leader $leading_space]] \n
            } else {
                dict incr undefined_references $chunk_id
            }
        } else {
            append result $leader $line \n
        }
    }
    # Trim off any trailing whitespace. It will be added back as the
    # chunk references are gather together. This avoids adding gratuitous
    # white space.
    return [string trimright $result]
}

Here we implement the logic around the -strict option using the accumulated data about defined and undefined references.

proc validateTangle {} {
    variable undefined_references
    variable chunk_references

    set unreferenced [dict filter $chunk_references value 0]
    log::info "unreferenced chunks: '[join [dict keys $unreferenced] {, }]'"
    set undefined [dict keys $undefined_references]
    log::info "undefined chunks: '[join $undefined {, }]'"

    if {$::options(strict)} {
        set errors 0
        if {[llength $unreferenced] != 0} {
            incr errors
            log::error "unreferenced chunks: [join [dict keys $unreferenced] {, }]"
        }
        if {[llength $undefined] != 0} {
            incr errors
            log::error "undefined chunks: [join $undefined {, }]"
        }

        if {$errors > 0} {
            throw INVALID_CHUNKS "failed strict chunk consistency check"
        }
    }
}

Using the same data about chunk references accumulated during code assembly, we can print information about any possible inconsistencies between chunk definitions and chunk references.

proc summarize {} {
    if {!$::options(summary)} {
        return
    }

    variable undefined_references
    variable chunk_references

    set unreferenced [dict filter $chunk_references value 0]
    if {[dict size $unreferenced] == 0} {
        chan puts "All defined program chunks were referenced."
    } else {
        chan puts "Chunks that were defined but not referenced:"
        chan puts "  [join [dict keys $unreferenced] {, }]"
    }

    set undefined [dict keys $undefined_references]
    if {[llength $undefined] == 0} {
        chan puts "No undefined chunks were referenced."
    } else {
        chan puts "Chunks that were referenced but not defined:"
        chan puts "  [join $undefined {, }]"
    }
}

Code Layout

Dependencies

Since pantangle is a Tcl script, it is necessary to have an installation of Tcl and the Tcl standard library. This development was done with Tcl version 8.6.11 and Tcl library 1.20. Major Linux distributions and other package management systems distribute current versions of Tcl and Tcllib. See the Tcl website for access to the latest software releases.

The pantangle Namespace

It is common practice to place the main code logic in a separate namespace to avoid polluting the global namespace.

namespace eval ::pantangle {
    <<pantangle-variables>>
    <<logging-setup>>
    <<running-pandoc>>
    <<parsing-ast>>
    <<tangling-code>>
    <<building-chunk-graph>>
    <<validating-chunk-graph>>
    <<dot-graph>>
    <<summarize>>
    <<main>>
}

Program Code Root Chunk

Finally, we put together the tangle root of the pantangle program. This is where the assembly of the code for the program begins and all the code in pantangle is referenced, perhaps transitively, from this root.

#!/usr/bin/env tclsh

# This software is copyrighted 2025 by G. Andrew Mangogna.
# The following terms apply to all files associated with the software unless
# explicitly disclaimed in individual files.
# 
# The authors hereby grant permission to use, copy, modify, distribute,
# and license this software and its documentation for any purpose, provided
# that existing copyright notices are retained in all copies and that this
# notice is included verbatim in any distributions. No written agreement,
# license, or royalty fee is required for any of the authorized uses.
# Modifications to this software may be copyrighted by their authors and
# need not follow the licensing terms described here, provided that the
# new terms are clearly indicated on the first page of each file where
# they apply.
# 
# IN NO EVENT SHALL THE AUTHORS OR DISTRIBUTORS BE LIABLE TO ANY PARTY FOR
# DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
# OUT OF THE USE OF THIS SOFTWARE, ITS DOCUMENTATION, OR ANY DERIVATIVES
# THEREOF, EVEN IF THE AUTHORS HAVE BEEN ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
# 
# THE AUTHORS AND DISTRIBUTORS SPECIFICALLY DISCLAIM ANY WARRANTIES,
# INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT.  THIS SOFTWARE
# IS PROVIDED ON AN "AS IS" BASIS, AND THE AUTHORS AND DISTRIBUTORS HAVE
# NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS,
# OR MODIFICATIONS.
# 
# GOVERNMENT USE: If you are acquiring this software on behalf of the
# U.S. government, the Government shall have only "Restricted Rights"
# in the software and related documentation as defined in the Federal
# Acquisition Regulations (FARs) in Clause 52.227.19 (c) (2).  If you
# are acquiring the software on behalf of the Department of Defense,
# the software shall be classified as "Commercial Computer Software"
# and the Government shall have only "Restricted Rights" as defined in
# Clause 252.227-7013 (c) (1) of DFARs.  Notwithstanding the foregoing,
# the authors grant the U.S. Government and others acting in its behalf
# permission to use and distribute the software in accordance with the
# terms specified in this license.

set ::version 1.0.1

package require Tcl 8.6 9
<<required-packages>>
<<command-options>>
<<command-line-parsing>>
<<argument-validation>>
<<pantangle-namespace>>

::pantangle::main

For the case of pantangle, the graph of the tangled root is a simple tree. There are four chunks that handle the setup of the command line interface and the one chunk to hold the application logic in a separate namespace.

Introduction