= HIGHLIGHT PLUGIN MANUAL
André Simon
v3.59, October 2020
:lang: en
:toc: left
:toc-title: Contents
:toclevels: 4
:sectnums:
:sectnumlevels: 2
:sectanchors:
// Cross References:
:xrefstyle: short
:section-refsig: Sect.
// Misc Settings:
:experimental: true
:icons: font
:linkattrs: true
// GitHub Settings to enable Admonitions Icons in preview:
ifdef::env-github[]
:caution-caption: :fire:
:important-caption: :heavy_exclamation_mark:
:note-caption: :information_source:
:tip-caption: :bulb:
:warning-caption: :warning:
endif::[]

// =====================================
// Custom Attributes for Reference Links
// =====================================
// Folders:
:plugins: pass:q[link:plugins/[`plugins/`^]]
// Plugins files:
:bash_functions_lua: pass:q[link:plugins/bash_functions.lua[`bash_functions.lua`^]]
:cpp_ref_cplusplus_com_lua: pass:q[link:plugins/cpp_ref_cplusplus_com.lua[`cpp_ref_cplusplus_com.lua`^]]
:ctags_html_tooltips_lua: pass:q[link:plugins/ctags_html_tooltips.lua[`ctags_html_tooltips.lua`^]]
:outhtml_codefold_lua: pass:q[link:plugins/outhtml_codefold.lua[`outhtml_codefold.lua`^]]
:outhtml_curly_brackets_matcher_lua: pass:q[link:plugins/outhtml_curly_brackets_matcher.lua[`outhtml_curly_brackets_matcher.lua`^]]
:outhtml_keyword_matcher_lua: pass:q[link:plugins/outhtml_keyword_matcher.lua[`outhtml_keyword_matcher.lua`^]]
:outhtml_keyword_matcher_lua: pass:q[link:plugins/outhtml_keyword_matcher.lua[`outhtml_keyword_matcher.lua`^]]
:theme_invert_lua: pass:q[link:plugins/theme_invert.lua[`theme_invert.lua`^]]

== ABOUT

The plug-in interface allows modifications of syntax parsing, colouring and
the document's header and footer.

A common task would be to define a new set of syntax keywords, or to add items
to existing keyword groups. More advanced plug-ins add tooltips based on ctags
input or source code folding to the output (see <<EXAMPLES>>).


== SCRIPT STRUCTURE

The following script contains the basic plug-in structure which has no effect
on the output:

[source,lua]
--------------------------------------------------------------------------------
Description="Plugin Boilerplate"

-- optional parameter desc: syntax description
function syntaxUpdate(desc)
end

function themeUpdate(desc)
end

function formatUpdate(desc)
end

Plugins={
  { Type="theme", Chunk=themeUpdate },
  { Type="lang", Chunk=syntaxUpdate },
  { Type="format", Chunk=formatUpdate },
}
--------------------------------------------------------------------------------

The first line contains a description which gives a short summary of the
plug-in effects.

The next lines contain function definitions:

* The `syntaxUpdate` function is applied on syntax definition scripts (`+*.lang+`),
  whereas the `themeUpdate` function is applied on colour themes (`+*.theme+`).
* The `formatUpdate` is not executed in a syntax or theme Lua state.
  It just returns strings to override or enhance the document header and footer.
* The `desc` parameter contains the description string of the loaded syntax
  definition or colour theme. This information can be used to restrict
  modifications to certain kinds of input (i.e. only C code should be
  enhanced with syslog keywords).

Finally the `Plugins` array connects the functions to the Lua states which
are created when highlight loads a lang or theme script.
In this example, `themeUpdate` is connected to the theme state, and `syntaxUpdate`
to the lang state.
It is not required to always define both connections if your plugin only affects
one of the config script types.


== SYNTAX CHUNK ELEMENTS

The following list includes all variables and constants you may use to adjust
the parser's syntax highlighting.

Syntax definition items:

[horizontal]
`Comments`                :: table
`Description`             :: string
`Digits`                  :: string
`EnableIndentation`       :: boolean
`Identifiers`             :: string
`IgnoreCase`              :: boolean
`Keywords`                :: table
`NestedSections`          :: table
`Operators`               :: string
`PreProcessor`            :: table
`Strings`                 :: table


Document modification items:

[horizontal]
`HeaderInjection`         :: string
`FooterInjection`         :: string

Read only (internal highlighting states):

[horizontal]
`HL_STANDARD`             :: number
`HL_BLOCK_COMMENT`        :: number
`HL_BLOCK_COMMENT_END`    :: number
`HL_EMBEDDED_CODE_BEGIN`  :: number
`HL_EMBEDDED_CODE_END`    :: number
`HL_ESC_SEQ`              :: number
`HL_ESC_SEQ_END`          :: number
`HL_IDENTIFIER_BEGIN`     :: number
`HL_IDENTIFIER_END`       :: number
`HL_KEYWORD`              :: number
`HL_KEYWORD_END`          :: number
`HL_LINENUMBER`           :: number
`HL_LINE_COMMENT`         :: number
`HL_LINE_COMMENT_END`     :: number
`HL_NUMBER`               :: number
`HL_OPERATOR`             :: number
`HL_OPERATOR_END`         :: number
`HL_PREPROC`              :: number
`HL_PREPROC_END`          :: number
`HL_PREPROC_STRING`       :: number
`HL_STRING`               :: number
`HL_STRING_END`           :: number
`HL_UNKNOWN`              :: number
`HL_REJECT`               :: number

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number (selected format)
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Read only (other):

[horizontal]
`HL_PLUGIN_PARAM`         :: string (set with `--plug-in-param`)
`HL_LANG_DIR`             :: string (path of language definition directory)

Functions:

[horizontal]
`AddKeyword`              :: function
`RemoveKeyword`           :: function
`OnStateChange`           :: function
`Decorate`                :: function
`DecorateLineBegin`       :: function
`DecorateLineEnd`         :: function
`StoreValue`              :: function

[IMPORTANT]
================================================================================
Functions will only be executed if they are defined as local functions within the
`lang` chunk referenced in the `Plugins` array.
They will be ignored when defined elsewhere in the script.
================================================================================

The functions `AddKeyword`, `RemoveKeyword` and `OnStateChange` are also useful
in language definitions without a plug-in use case.


=== FUNCTIONS ADDKEYWORD/REMOVEKEYWORD

[[AddKeyword]]
The `AddKeyword` function will add a keyword to one of the the internal keyword
lists. It has no effect if the keyword was added before.
Keywords added with `AddKeyword` will remain active for all files of the same
syntax if highlight is in batch mode.

................................................................................
AddKeyword(keyword, kwGroupID)

  Parameters: keyword:   string which should be added to a keyword list
              kwGroupID: keyword group ID of the keyword
  Returns:    true if successful
................................................................................

[[RemoveKeyword]]
The `RemoveKeyword` function erases the given keyword from the internal list.

................................................................................
RemoveKeyword(keyword)

  Parameters: keyword:   string which should be removed from the keyword list
  Returns:    true if successful
................................................................................


=== FUNCTION ADDPERSISTENTSTATE

[[AddPersistentState]]
This function enables storage of keywords and keyword ranges in a plug-in file.
If the syntax contains elements which depend on a context, you can highlight
them although this context is lost in other input files or code sections.
The invocation of AddPersistentState will cause highlight to save a plugin as
temporary file and parse input files using this plug-in again if necessary.

................................................................................
AddPersistentState(keyword, kwGroupID)

  Parameters: keyword:   string which should be added to a keyword list
              kwGroupID: keyword group ID of the keyword
  Returns:    true if successful
................................................................................

................................................................................
AddPersistentState(lineno, kwGroupID, column, length)

  Parameters: lineno:    line number
              kwGroupID: the keyword group ID
              column:    column
              length:    length of the keyword
  Returns:    true if successful
................................................................................


=== FUNCTION ONSTATECHANGE

[[OnStateChange]]
This function is a hook which is called if an internal state changes (e.g. from
`HL_STANDARD` to `HL_KEYWORD` if a keyword is found). It can be used to alter
the new state or to manipulate syntax elements like keyword lists.

................................................................................
OnStateChange(oldState, newState, token, kwGroupID, lineno, column)

  Hook Event: Highlighting parser state change
  Parameters: oldState:  old state
              newState:  intended new state
              token:     the current token which triggered the new state
              kwGroupID: if newState is HL_KEYWORD, the parameter
                         contains the keyword group ID
              lineno:    line number (since 3.50)
              column:    line column (since 3.50)
  Returns:    Correct state to continue OR HL_REJECT
................................................................................

Returns `HL_REJECT` if the recognized token and state should be discarded;
the first character of token will be outputted and highlighted as `oldState`.


Example for its usage in a plugin context:

[source,lua]
--------------------------------------------------------------------------------
function OnStateChange(oldState, newState, token, kwgroup)
   if newState==HL_KEYWORD and kwgroup==5 then
      AddKeyword(token, 5)
   end
   return newState
end
--------------------------------------------------------------------------------

This function adds the current keyword to the internal keyword list if the
keyword belongs to keyword group 5. If keyword group 5 is defined by a regex,
this token will be recognized later as a keyword even if the regular expression
does no longer match.

[IMPORTANT]
================================================================================
If both the syntax file and the plug-in script contain an `OnStateChange`
function, the plug-in definition will replace the function of the syntax file.
In order to prevent this, the existing function can be assigned to an new
variable and called using this name in the plug-in `OnStateChange` code chunk.

See this example in the plug-in `bash_functions.lua`:
================================================================================

[source,lua]
--------------------------------------------------------------------------------
function syntaxUpdate(desc)
  if desc=="Bash" then

  table.insert( Keywords,
                  { Id=5, Regex=[[(\w+)\s*\(]]
                  } )

    if OnStateChange ~= nil then
      OrigOnStateChange = OnStateChange;
    end

    -- add keywords to list 5 if pattern matches
    function OnStateChange(oldState, newState, token, kwgroup)

      if newState==HL_KEYWORD and kwgroup==5 then
        AddKeyword(token, 5)
        return newState
      end
      if OrigOnStateChange then
        return OrigOnStateChange(oldState, newState, token, kwgroup)
      end
    end

  end
end
--------------------------------------------------------------------------------


=== FUNCTION OVERRIDEPARAM

[[OverrideParam]]
This function enables modification of internal default values concerning the
parser or output formats. This function is experimental and subject to change.
Currently it supports these parameters:

[horizontal]
`state.string.raw`    :: `true` or `false`
`format.maskws`       :: `true` or `false`
`format.spacer`       :: whitespace format string

See `langDefs/toml.lang` and `plugins/outhtml_ie7_webctrl.lua` for examples.

................................................................................
OverrideParam(keyword, kwGroupID)

  Parameters: paramName:   name of configuration item
              paramVal:    value
  Returns:    true if successful
................................................................................


=== DECORATE FUNCTIONS

[[Decorate]]
The `Decorate` function is a hook which is called if a syntax token has been
identified. It can be used to alter the token or to add additional text in the
target output format (e.g. hyperlinks).

................................................................................
Decorate(token, state, kwGroupID, lineHasStmt, lineno, column)

  Hook Event: Token identification
  Parameters: token:       current token
              state:       current state
              kwGroupID:   if state is HL_KEYWORD, the parameter
                           contains the keyword group ID
              lineHasStmt: true if current line contains a statement
              lineno:    line number (since 3.60)
              column:    line column (since 3.60)
  Returns:    Altered token string or nothing if original token should be
              outputted
................................................................................

Examples:

[source,lua]
--------------------------------------------------------------------------------
function Decorate(token, state)
  if (state == HL_KEYWORD) then
    return string.upper(token)
  end
end
--------------------------------------------------------------------------------

This function converts all keywords to upper case.

The functions `DecorateLineBegin` and `DecorateLineEnd` are called if a new line
starts or ends. They can be used to add special formatting to lines of code.

[[DecorateLineBegin]]
................................................................................
DecorateLineBegin(lineNumber)

  Hook Event: output of a new line
  Parameters: lineNumber: the current line number
  Returns:    A string to be prepended to a new line (or nothing)
................................................................................

[[DecorateLineEnd]]
................................................................................
DecorateLineEnd(lineNumber)

  Hook Event: output of a line ending
  Parameters: lineNumber: the current line number
  Returns:    A string to be appended to a line (or nothing)
................................................................................


[IMPORTANT]
================================================================================
The return value of `Decorate` functions will be embedded in the formatting tags
of the output format.
The return values are not modified or validated by highlight.
================================================================================

=== FUNCTION STOREVALUE

[[StoreValue]]
................................................................................
StoreValue(name, value)

  Set or get a value to exchange information across Lua states
  Parameters: name: the parameter name
              value: the parameter value
  Returns:   if value is not set, the previously assigned value of name
................................................................................


== THEME CHUNK ELEMENTS

The following list includes all those items which you can overwrite or extend to
adjust the formatting (colour and font attributes) of the output:

Output formatting items:

[horizontal]
`Default`                 :: table
`Canvas`                  :: table
`Number`                  :: table
`Escape`                  :: table
`String`                  :: table
`StringPreProc`           :: table
`BlockComment`            :: table
`PreProcessor`            :: table
`LineNum`                 :: table
`Operator`                :: table
`LineComment`             :: table
`Keywords`                :: table

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Add additional styling information:

[horizontal]
`Injections`              :: table

[horizontal]
`StoreValue`              :: function


=== FUNCTION STOREVALUE

[[StoreValue]]
................................................................................
Like above.
................................................................................


== FORMAT CHUNK ELEMENTS

Read only (output document format):

[horizontal]
`HL_OUTPUT`               :: number
`HL_FORMAT_HTML`          :: number
`HL_FORMAT_XHTML`         :: number
`HL_FORMAT_TEX`           :: number
`HL_FORMAT_LATEX`         :: number
`HL_FORMAT_RTF`           :: number
`HL_FORMAT_ANSI`          :: number
`HL_FORMAT_XTERM256`      :: number
`HL_FORMAT_TRUECOLOR`     :: number
`HL_FORMAT_SVG`           :: number
`HL_FORMAT_BBCODE`        :: number
`HL_FORMAT_PANGO`         :: number
`HL_FORMAT_ODT`           :: number

Functions:

[horizontal]
`DocumentHeader`          :: function
`DocumentFooter`          :: function


=== DOCUMENTHEADER FUNCTION

[[DocumentHeader]]
................................................................................
DocumentHeader(numFiles, currFile, options)

  Hook Event: output of a new file's header
  Parameters: numFiles: number of files to be generated
              currFile: current file counter
              options: Map of the following options
              options.title: document title
              options.encoding: document encoding
              options.fragment: true if header/footer should not be outputted
              options.font: font name
              options.fontsize: font size

  Returns:    [string, boolean?] (or nothing)
              The string contains the new document header
              The boolean value indicates if the string should replace the default
              header (false=default) or if it should be appended to it (true).
................................................................................


=== DOCUMENTFOOTER FUNCTION

[[DocumentFooter]]
................................................................................
DocumentFooter(numFiles, currFile, options)

  Hook Event: output of a new file's footer
  Parameters: see DocumentHeader

  Returns:    [string, boolean?] (or nothing)
              The string contains the new document footer
              The boolean value indicates if the string should replace the default
              footer (false=default) or if it should precede it (true).
................................................................................


== STEP BY STEP

This example will add reference hyperlinks to Qt keywords:

[source,lua]
--------------------------------------------------------------------------------
-- first add a description of what the plug-in does
Description="Add qtproject.org reference links to HTML, LaTeX or RTF output"

-- define the plugin categories (ie. supported output formats; languages)
Categories = {"c++", "qt" }

-- the syntaxUpdate function contains code related to syntax recognition
function syntaxUpdate(desc)

  -- if the current file is no C++ file we exit
  if desc~="C and C++" then
     return
  end

  -- this function returns a qt-project reference link of the given token
  function getURL(token)
     -- generate the URL
     url='http://qt-project.org/doc/qt-4.8/'..string.lower(token).. '.html'

     -- embed the URL in a hyperlink according to the output format
     -- first HTML, then LaTeX and RTF
     if (HL_OUTPUT== HL_FORMAT_HTML or HL_OUTPUT == HL_FORMAT_XHTML) then
         return '<a class="hl" target="new" href="'
                .. url .. '">'.. token .. '</a>'
     elseif (HL_OUTPUT == HL_FORMAT_LATEX) then
         return '\\href{'..url..'}{'..token..'}'
     elseif (HL_OUTPUT == HL_FORMAT_RTF) then
         return '{{\\field{\\*\\fldinst HYPERLINK "'
                ..url..'" }{\\fldrslt\\ul\\ulc0 '..token..'}}}'
     end
   end

  -- the Decorate function will be invoked for every recognized token
  function Decorate(token, state)

    -- we are only interested in keywords, preprocessor or default items
    if (state ~= HL_STANDARD and state ~= HL_KEYWORD and
        state ~=HL_PREPROC) then
      return
    end

    -- Qt keywords start with Q, followed by an upper and a lower case letter
    -- if this pattern applies to the token, we return the URL
    -- if we return nothing, the token is outputted as is
    if string.find(token, "Q%u%l")==1 then
      return getURL(token)
    end

  end
end

-- the themeUpdate function contains code related to the theme
function themeUpdate(desc)
  -- the Injections table can be used to add style information to the theme

  -- HTML: we add additional CSS style information to beautify hyperlinks,
  -- they should have the same color as their surrounding tags
  if (HL_OUTPUT == HL_FORMAT_HTML or HL_OUTPUT == HL_FORMAT_XHTML) then
    Injections[#Injections+1]=
      "a.hl, a.hl:visited {color:inherit;font-weight:inherit;}"

  -- LaTeX: hyperlinks require the hyperref package, so we add this here
  -- the colorlinks and pdfborderstyle options remove ugly boxes in the output
  elseif (HL_OUTPUT==HL_FORMAT_LATEX) then
    Injections[#Injections+1]=
      "\\usepackage[colorlinks=false, pdfborderstyle={/S/U/W 1}]{hyperref}"
  end
end

-- let highlight load the chunks
Plugins={
  { Type="lang", Chunk=syntaxUpdate },
  { Type="theme", Chunk=themeUpdate },
}
--------------------------------------------------------------------------------


== EXAMPLES


The {plugins} directory contains example scripts, including:

{bash_functions_lua}::
+
[horizontal]
Description ::: Add function names to keyword list.
Features    ::: Adds new keyword group based on a regex, defines `OnStateChange`,
                uses `AddKeyword`.

{theme_invert_lua}::
+
[horizontal]
Description ::: Invert colours of the original theme.
Features    ::: Modifies all color attributes of the theme script, uses Lua
                pattern matching.

{cpp_ref_cplusplus_com_lua}::
+
[horizontal]
Description ::: Add qtproject.org reference links to HTML, LaTeX or RTF output of
                C++ code.
Features    ::: Uses `Decorate` to add hyperlinks for a defined set of C++ keywords. +
                Adds CSS styles with `Injections`.

{ctags_html_tooltips_lua}::
+
[horizontal]
Description ::: Add tooltips based on a ctags file (default input file: tags).
Features    ::: Uses file input (defined by cli option `--plug-in-param`) and
                parses tags data before `Decorate` is called.

{outhtml_curly_brackets_matcher_lua}::
+
[horizontal]
Description ::: Shows matching curly brackets in HTML output.
Features    ::: Uses `Decorate` to add span tags with unique ids to opening and
                closing brackets. +
                Adds JavaScript with `HeaderInjection` variable. +
                Inserts additional CSS styles with `Injections` variable.

{outhtml_keyword_matcher_lua}::
+
[horizontal]
Description ::: Shows matching keywords in HTML output.
Features    ::: Uses `Decorate` to add span tags with unique ids to keywords. +
                Uses `OnStateChange` to assign an internal ID to each keyword. +
                Adds JavaScript with `HeaderInjection` variable. +
                Inserts additional CSS styles with `Injections` variable. +

{outhtml_codefold_lua}::
+
[horizontal]
Description ::: Adds code folding for C style languages, Pascal, Lua and Ruby to
                HTML output.
Features    ::: Uses `DecorateLineBegin` and `DecorateLineEnd` to add ID-spans to each
                line. +
                Applies `Decorate` to each code block delimiter to add `onClick` event
                handlers. +
                Adds JavaScript with `HeaderInjection` and `FooterInjection` variables. +
                Inserts additional CSS styles with `Injections` variable.


== PLUG-IN USAGE

[discrete]
=== Command line interface

Run highlight `--list-scripts=plugins` to show all installed plug-ins.

Use `--plug-in` to load a plug-in script file.
This option can be applied more than once to apply several plug-ins.
Omit the `.lua` suffix.
You can store your plug-in scripts for testing in `~/.highlight/plugins`.

.Example

................................................................................
highlight my.cpp -Ilz --plug-in=html_curly_brackets_matcher > ~/test_out/my.html
................................................................................


[discrete]
=== GUI

Add the plug-in scripts in the plug-in selection tab and enable them using the
checkboxes.


// EOF //
