Tag version 0.7.0.

release.md: Add details about a step.
tests: Fix the last added test to pass when sudo(8) isn't installed.
2020-02-28 15:36:43 +00:00 · 2020-02-28 15:36:25 +00:00 · 2020-02-28 15:22:43 +00:00 · 2020-02-25 17:34:35 +00:00 · 2020-02-25 17:22:55 +00:00 · 2020-02-25 07:37:09 -06:00
47 changed files with 1365 additions and 58 deletions
--- a/.editorconfig
+++ b/.editorconfig
@@ -0,0 +1,9 @@
+# Top-most editorconfig file
+
+root = true
+
+[*]
+end_of_line = lf
+tab_width   = 2
+indent_size = 2
+indent_style = space
--- a/.version
+++ b/.version
@@ -1 +1 @@
-0.7.0-dev
+0.7.0
--- a/COPYING.md
+++ b/COPYING.md
@@ -1,4 +1,4 @@
-Copyright (c) 2010-2018 zsh-syntax-highlighting contributors
+Copyright (c) 2010-2020 zsh-syntax-highlighting contributors
 All rights reserved.

 Redistribution and use in source and binary forms, with or without modification, are permitted
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -6,7 +6,7 @@ How to install
 * Arch Linux: [community/zsh-syntax-highlighting][arch-package] / [AUR/zsh-syntax-highlighting-git][AUR-package]
 * Debian: `zsh-syntax-highlighting` package [in `stretch`][debian-package] (or in [OBS repository][obs-repository])
 * Fedora: [zsh-syntax-highlighting package][fedora-package-alt] in Fedora 24+ (or in [OBS repository][obs-repository])
-* FreeBSD: `pkg install zsh-syntax-highlighting` (port name: [`textproc/zsh-syntax-highlighting`][freebsd-port])
+* FreeBSD: `pkg install zsh-syntax-highlighting` (port name: [`shells/zsh-syntax-highlighting`][freebsd-port])
 * Gentoo: [mv overlay][gentoo-overlay]
 * Mac OS X / Homebrew: [brew install zsh-syntax-highlighting][brew-package]
 * Ubuntu: `zsh-syntax-highlighting` package [in Xenial][ubuntu-package] (or in [OBS repository][obs-repository])
--- a/changelog.md
+++ b/changelog.md
@@ -1,3 +1,84 @@
+# Changes in version 0.7.0
+
+**Version 0.7.0 has not been released.  This changelog is a work in progress.**
+
+This is a stable bugfix and feature release.  Major new features and changes include:
+
+- Add `ZSH_HIGHLIGHT_DIRS_BLACKLIST` to disable "path" and "path prefix"
+  highlighting for specific directories
+  [#379]
+
+- Add the "regexp" highlighter, modelled after the pattern highlighter
+  [4e6f60063f1c]
+
+- When a word uses globbing, only the globbing metacharacters will be highlighted as globbing:
+  in `: foo*bar`, only the `*` will be blue.
+  [e48af357532c]
+
+- Highlight pasted quotes (e.g., `: foo"bar"`)
+  [dc1b2f6fa4bb]
+
+- Highlight command substitutions (`` : `ls` ``, `: $(ls)`)
+  [c0e64fe13178 and parents, e86f75a840e7, et al]
+
+- Highlight process substitutions (`: >(nl)`, `: <(pwd)`, `: =(git diff)`)
+  [c0e64fe13178 and parents, e86f75a840e7, et al]
+
+- Highlight command substitutions inside double quotes (``: "`foo`"``)
+  [f16e858f0c83]
+
+- Highlight many precommands (e.g., `nice`, `stdbuf`, `eatmydata`;
+  see `$precommand_options` in the source)
+
+- Highlight numeric globs (e.g., `echo /lib<->`)
+
+- Assorted improvement to aliases highlighting
+  (e.g.,
+   `alias sudo_u='sudo -u'; sudo_u jrandom ls`,
+   `alias x=y y=z z=nosuchcommand; x`,
+   `alias ls='ls -l'; \ls`)
+  [f3410c5862fc, 57386f30aec8, #544, and many others]
+
+- Highlight some more syntax errors
+  [dea05e44e671, 298ef6a2fa30]
+
+- New styles: named file descriptors, `RC_QUOTES`, and unclosed quotes (e.g., `echo "foo<CURSOR>`)
+  [38c794a978cd, 25ae1c01216c, 967335dfc5fd]
+
+- The 'brackets' highlighting no longer treats quotes specially.
+  [ecdda36ef56f]
+
+
+Selected bugfixes include:
+
+- Highlight `sudo` correctly when it's not installed
+  [26a82113b08b]
+
+- Handle some non-default options being set in zshrc
+  [b07ada1255b7, a2a899b41b8, 972ad197c13d, b3f66fc8748f]
+
+- Fix off-by-one highlighting in vi "visual" mode (vicmd keymap)
+  [be3882aeb054]
+
+- The 'yank-pop' widget is not wrapped
+  [#183]
+
+
+Known issues include:
+
+- A multiline alias that uses a simple command terminator (such as `;`, `|`, `&&`)
+  before a newline will incorrectly be highlighted as an error.  See issue #677
+  for examples and workarounds.
+  [#677]
+
+- Use of a simple parameter expansion (`${foo}` with nothing but a parameter
+  name inside the braces) in command position, when the value of the parameter
+  looks like an assignment (such as after `foo='bar=$(ls)'`), may result in
+  incorrect highlighting and "BUG" messages.  As a workaround, the expansion
+  may be nested (`${${foo}}`).
+  [#670]
+
+
 # Changes in version 0.6.0

 This is a stable release, featuring bugfixes and minor improvements.
--- a/docs/highlighters.md
+++ b/docs/highlighters.md
@@ -104,7 +104,7 @@ To create your own `acme` highlighter:
        `_zsh_highlight_highlighter_acme_paint` respectively.

        These names are still supported for backwards compatibility;
-        however, support for them will be removed in a a future major or minor release (v0.x.0 or v1.0.0).
+        however, support for them will be removed in a future major or minor release (v0.x.0 or v1.0.0).

 * Activate your highlighter in `~/.zshrc`:

--- a/docs/highlighters/main.md
+++ b/docs/highlighters/main.md
@@ -86,7 +86,6 @@ manual page][zshzle-Character-Highlighting].
 #### Parameters

 To avoid partial path lookups on a path, add the path to the `ZSH_HIGHLIGHT_DIRS_BLACKLIST` array.
-This interface is still experimental.

 ```zsh
 ZSH_HIGHLIGHT_DIRS_BLACKLIST+=(/mnt/slow_share)
--- a/highlighters/main/main-highlighter.zsh
+++ b/highlighters/main/main-highlighter.zsh
@@ -1,5 +1,5 @@
 # -------------------------------------------------------------------------------------------------
-# Copyright (c) 2010-2018 zsh-syntax-highlighting contributors
+# Copyright (c) 2010-2020 zsh-syntax-highlighting contributors
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without modification, are permitted
@@ -77,6 +77,16 @@ _zsh_highlight_main_add_region_highlight() {
    [[ $1 == unknown-token ]] && alias_style=unknown-token
    return
  fi
+  if (( in_param )); then
+    if [[ $1 == unknown-token ]]; then
+      param_style=unknown-token
+    fi
+    if [[ -n $param_style ]]; then
+      return
+    fi
+    param_style=$1
+    return
+  fi

  # The calculation was relative to $buf but region_highlight is relative to $BUFFER.
  (( start += buf_offset ))
@@ -188,7 +198,7 @@ _zsh_highlight_main__type() {
    elif {  [[ $1 != */* ]] || is-at-least 5.3 } &&
         # Add a subshell to avoid a zsh upstream bug; see issue #606.
         # ### Remove the subshell when we stop supporting zsh 5.7.1 (I assume 5.8 will have the bugfix).
-         ! (builtin type -w -- $1) >/dev/null 2>&1; then
+         ! (builtin type -w -- "$1") >/dev/null 2>&1; then
      REPLY=none
    fi
  fi
@@ -203,7 +213,7 @@ _zsh_highlight_main__type() {
    # starts with an arithmetic expression [«((…))» as the first thing inside
    # «$(…)»], which is area that has had some parsing bugs before 5.6
    # (approximately).
-    REPLY="${$(:; (( aliases_allowed )) || unalias -- $1 2>/dev/null; LC_ALL=C builtin type -w -- $1 2>/dev/null)##*: }"
+    REPLY="${$(:; (( aliases_allowed )) || unalias -- "$1" 2>/dev/null; LC_ALL=C builtin type -w -- "$1" 2>/dev/null)##*: }"
    if [[ $REPLY == 'alias' ]]; then
      may_cache=0
    fi
@@ -309,20 +319,33 @@ _zsh_highlight_highlighter_main_paint()
    'builtin' ''
    'command' :pvV
    'exec' a:cl
-    'nocorrect' ''
    'noglob' ''
-    'time' ''
+    # 'time' and 'nocorrect' shouldn't be added here; they're reserved words, not precommands.

    'doas' aCu:Lns # as of OpenBSD's doas(1) dated September 4, 2016
    'nice' n: # as of current POSIX spec
    'pkexec' '' # doesn't take short options; immune to #121 because it's usually not passed --option flags
-    'sudo' Cgprtu:AEHKPSVbhiklnsv # as of sudo 1.8.21p2
+    # Argumentless flags that can't be followed by a command: -e -h -K -k -V -v
+    'sudo' Cgprtu:AEHPSbilns # as of sudo 1.8.21p2
    'stdbuf' ioe:
    'eatmydata' ''
    'catchsegv' ''
    'nohup' ''
    'setsid' :wc
+    # As of OpenSSH 8.1p1; -k is deliberately left out since it may not be followed by a command
+    'ssh-agent' aEPt:csDd
+    # suckless-tools v44
+    # Argumentless flags that can't be followed by a command: -v
+    'tabbed' gnprtTuU:cdfhs
+
+    # moreutils 0.62-1
+    'chronic' :ev
+    'ifne' :n
+
  )
+  # Commands that would need to skip one positional argument:
+  #    flock
+  #    ssh

  if [[ $zsyh_user_options[ignorebraces] == on || ${zsyh_user_options[ignoreclosebraces]:-off} == on ]]; then
    local right_brace_is_recognised_everywhere=false
@@ -393,11 +416,13 @@ _zsh_highlight_main_highlighter_highlight_list()
  # alias_style is the style to apply to an alias once in_alias=0
  #     Usually 'alias' but set to 'unknown-token' if any word expanded from
  #     the alias would be highlighted as unknown-token
-  local alias_style arg buf=$4 highlight_glob=true style
+  # param_style is analogous for parameter expansions
+  local alias_style param_style arg buf=$4 highlight_glob=true style
  local in_array_assignment=false # true between 'a=(' and the matching ')'
  # in_alias is equal to the number of shifts needed until arg=args[1] pops an
  #     arg from BUFFER and not added by an alias.
-  integer in_alias=0 len=$#buf
+  # in_param is analogous for parameter expansions
+  integer in_alias=0 in_param=0 len=$#buf
  local -a match mbegin mend list_highlights
  # seen_alias is a map of aliases already seen to avoid loops like alias a=b b=a
  local -A seen_alias
@@ -433,12 +458,12 @@ _zsh_highlight_main_highlighter_highlight_list()
  #
  # When the kind of a word is not yet known, $this_word / $next_word may contain
  # multiple states.  For example, after "sudo -i", the next word may be either
-  # another --flag or a command name, hence the state would include both :start:
-  # and :sudo_opt:.
+  # another --flag or a command name, hence the state would include both ':start:'
+  # and ':sudo_opt:'.
  #
  # The tokens are always added with both leading and trailing colons to serve as
-  # word delimiters (an improvised array); [[ $x == *:foo:* ]] and x=${x//:foo:/}
-  # will DTRT regardless of how many elements or repetitions $x has..
+  # word delimiters (an improvised array); [[ $x == *':foo:'* ]] and x=${x//:foo:/}
+  # will DTRT regardless of how many elements or repetitions $x has.
  #
  # Handling of redirections: upon seeing a redirection token, we must stall
  # the current state --- that is, the value of $this_word --- for two iterations
@@ -476,6 +501,14 @@ _zsh_highlight_main_highlighter_highlight_list()
        _zsh_highlight_main_add_region_highlight $start_pos $end_pos $alias_style
      fi
    fi
+    if (( in_param )); then
+      (( in_param-- ))
+      if (( in_param == 0 )); then
+        # start_pos and end_pos are of the '$foo' word (previous $arg) here
+        _zsh_highlight_main_add_region_highlight $start_pos $end_pos $param_style
+        param_style=""
+      fi
+    fi

    # Initialize this_word and next_word.
    if (( in_redirection == 0 )); then
@@ -500,7 +533,7 @@ _zsh_highlight_main_highlighter_highlight_list()
      fi
    fi

-    if (( in_alias == 0 )); then
+    if (( in_alias == 0 && in_param == 0 )); then
      # Compute the new $start_pos and $end_pos, skipping over whitespace in $buf.
      [[ "$proc_buf" = (#b)(#s)(([ $'\t']|\\$'\n')#)* ]]
      # The first, outer parenthesis
@@ -543,7 +576,7 @@ _zsh_highlight_main_highlighter_highlight_list()
      continue
    fi

-    if [[ $this_word == *:start:* ]] && ! (( in_redirection )); then
+    if [[ $this_word == *':start:'* ]] && ! (( in_redirection )); then
      # Expand aliases.
      # An alias is ineligible for expansion while it's being expanded (see #652/#653).
      _zsh_highlight_main__type "$arg" "$(( ! ${+seen_alias[$arg]} ))"
@@ -578,8 +611,7 @@ _zsh_highlight_main_highlighter_highlight_list()
        continue
      else
        _zsh_highlight_main_highlighter_expand_path $arg
-        arg=$REPLY
-        _zsh_highlight_main__type "$arg" 0
+        _zsh_highlight_main__type "$REPLY" 0
        res="$REPLY"
      fi
    fi
@@ -614,6 +646,7 @@ _zsh_highlight_main_highlighter_highlight_list()
      local -a match mbegin mend
      local MATCH; integer MBEGIN MEND
      local parameter_name
+      local -a words
      if [[ $arg[1] == '$' ]] && [[ ${arg[2]} == '{' ]] && [[ ${arg[-1]} == '}' ]]; then
        parameter_name=${${arg:2}%?}
      elif [[ $arg[1] == '$' ]]; then
@@ -626,14 +659,16 @@ _zsh_highlight_main_highlighter_highlight_list()
        # Set $arg.
        case ${(tP)MATCH} in
          (*array*|*assoc*)
-            local -a words; words=( ${(P)MATCH} )
-            arg=${words[1]}
+            words=( ${(P)MATCH} )
            ;;
          (*)
            # scalar, presumably
-            arg=${(P)MATCH}
+            words=( ${(P)MATCH} )
            ;;
        esac
+        (( in_param = 1 + $#words ))
+        args=( $words $args )
+        arg=$args[1]
        _zsh_highlight_main__type "$arg" 0
        res=$REPLY
      fi
@@ -658,16 +693,20 @@ _zsh_highlight_main_highlighter_highlight_list()
        elif [[ -n $flags_sans_argument ]] &&
             [[ $arg == '-'[$flags_sans_argument]# ]]; then
          # Flag that requires no argument
-          this_word=:sudo_opt:
+          this_word=':sudo_opt:'
          next_word+=':start:'
          next_word+=':sudo_opt:'
        elif [[ $arg == '-'* ]]; then
-          # Unknown flag
-          this_word=:sudo_opt:
+          # Unknown flag.  We don't know whether it takes an argument or not,
+          # so modify $next_word as we do for flags that require no argument.
+          # With that behaviour, if the flag in fact takes no argument we'll
+          # highlight the inner command word correctly, and if it does take an
+          # argument we'll highlight the command word correctly if the argument
+          # was given in the same shell word as the flag (as in '-uphy1729' or
+          # '--user=phy1729' without spaces).
+          this_word=':sudo_opt:'
          next_word+=':start:'
          next_word+=':sudo_opt:'
-          _zsh_highlight_main_add_region_highlight $start_pos $end_pos unknown-token
-          continue
        else
          # Not an option flag; nothing to do.  (If the command line is
          # syntactically valid, ${this_word//:sudo_opt:/} should be
@@ -720,6 +759,10 @@ _zsh_highlight_main_highlighter_highlight_list()
                        style=reserved-word
                        # Match braces and handle special cases.
                        case $arg in
+                          (time|nocorrect)
+                            next_word=${next_word//:regular:/}
+                            next_word+=':start:'
+                            ;;
                          ($'\x7b')
                            braces_stack='Y'"$braces_stack"
                            ;;
@@ -797,7 +840,7 @@ _zsh_highlight_main_highlighter_highlight_list()
        function)       style=function;;
        command)        style=command;;
        hashed)         style=hashed-command;;
-        none)           if _zsh_highlight_main_highlighter_check_assign; then
+        none)           if (( ! in_param )) && _zsh_highlight_main_highlighter_check_assign; then
                          _zsh_highlight_main_add_region_highlight $start_pos $end_pos assign
                          local i=$(( arg[(i)=] + 1 ))
                          if [[ $arg[i] == '(' ]]; then
@@ -818,11 +861,14 @@ _zsh_highlight_main_highlighter_highlight_list()
                            fi
                          fi
                          continue
-                        elif [[ $arg[0,1] = $histchars[0,1] ]] && (( $#arg[0,2] == 2 )); then
+                        elif (( ! in_param )) &&
+                             [[ $arg[0,1] = $histchars[0,1] ]] && (( $#arg[0,2] == 2 )); then
                          style=history-expansion
-                        elif [[ $arg[0,1] == $histchars[2,2] ]]; then
+                        elif (( ! in_param )) &&
+                             [[ $arg[0,1] == $histchars[2,2] ]]; then
                          style=history-expansion
-                        elif [[ $arg[1,2] == '((' ]]; then
+                        elif (( ! in_param )) &&
+                             [[ $arg[1,2] == '((' ]]; then
                          # Arithmetic evaluation.
                          #
                          # Note: prior to zsh-5.1.1-52-g4bed2cf (workers/36669), the ${(z)...}
@@ -837,14 +883,17 @@ _zsh_highlight_main_highlighter_highlight_list()
                            _zsh_highlight_main_add_region_highlight $((end_pos - 2)) $end_pos reserved-word
                          fi
                          continue
-                        elif [[ $arg == '()' ]]; then
+                        elif (( ! in_param )) &&
+                             [[ $arg == '()' ]]; then
                          # anonymous function
                          style=reserved-word
-                        elif [[ $arg == $'\x28' ]]; then
+                        elif (( ! in_param )) &&
+                             [[ $arg == $'\x28' ]]; then
                          # subshell
                          style=reserved-word
                          braces_stack='R'"$braces_stack"
-                        elif [[ $arg == $'\x29' ]]; then
+                        elif (( ! in_param )) &&
+                             [[ $arg == $'\x29' ]]; then
                          # end of subshell or command substitution
                          if _zsh_highlight_main__stack_pop 'S'; then
                            REPLY=$start_pos
@@ -926,6 +975,7 @@ _zsh_highlight_main_highlighter_highlight_list()
    _zsh_highlight_main_add_region_highlight $start_pos $end_pos $style
  done
  (( in_alias == 1 )) && in_alias=0 _zsh_highlight_main_add_region_highlight $start_pos $end_pos $alias_style
+  (( in_param == 1 )) && in_param=0 _zsh_highlight_main_add_region_highlight $start_pos $end_pos $param_style
  [[ "$proc_buf" = (#b)(#s)(([[:space:]]|\\$'\n')#) ]]
  REPLY=$(( end_pos + ${#match[1]} - 1 ))
  reply=($list_highlights)
--- a/highlighters/main/test-data/abspath-in-command-position1.zsh
+++ b/highlighters/main/test-data/abspath-in-command-position1.zsh
@@ -0,0 +1,35 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'/'
+
+expected_region_highlight=(
+  '1 1 path' # /
+)
--- a/highlighters/main/test-data/abspath-in-command-position2.zsh
+++ b/highlighters/main/test-data/abspath-in-command-position2.zsh
@@ -0,0 +1,35 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'/bi'
+
+expected_region_highlight=(
+  '1 3 path_prefix' # /bi
+)
--- a/highlighters/main/test-data/abspath-in-command-position3.zsh
+++ b/highlighters/main/test-data/abspath-in-command-position3.zsh
@@ -0,0 +1,35 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'/bin'
+
+expected_region_highlight=(
+  '1 4 path' # /bin
+)
--- a/highlighters/main/test-data/abspath-in-command-position4.zsh
+++ b/highlighters/main/test-data/abspath-in-command-position4.zsh
@@ -0,0 +1,35 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'/bin/s'
+
+expected_region_highlight=(
+  '1 6 path_prefix' # /bin/s
+)
--- a/highlighters/main/test-data/abspath-in-command-position5.zsh
+++ b/highlighters/main/test-data/abspath-in-command-position5.zsh
@@ -0,0 +1,35 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'/bin/sh'
+
+expected_region_highlight=(
+  '1 7 command' # /bin/sh
+)
--- a/highlighters/main/test-data/alias-basic.zsh
+++ b/highlighters/main/test-data/alias-basic.zsh
@@ -0,0 +1,35 @@
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2015 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+alias foo="echo hello world"
+BUFFER="foo"
+
+expected_region_highlight+=(
+  "1 3 alias" # foo
+)
--- a/highlighters/main/test-data/alias-loop2.zsh
+++ b/highlighters/main/test-data/alias-loop2.zsh
@@ -0,0 +1,35 @@
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2015 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+alias ls="ls"
+BUFFER="ls"
+
+expected_region_highlight+=(
+  "1 2 alias" # ls
+)
--- a/highlighters/main/test-data/alias-parameter.zsh
+++ b/highlighters/main/test-data/alias-parameter.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+alias '$foo'='echo alias'
+local foo; foo=(echo param)
+
+BUFFER='$foo'
+
+expected_region_highlight=(
+  '1 4 alias' # $foo
+)
--- a/highlighters/main/test-data/alias-precommand-option-argument1.zsh
+++ b/highlighters/main/test-data/alias-precommand-option-argument1.zsh
@@ -28,6 +28,7 @@
 # vim: ft=zsh sw=2 ts=2 et
 # -------------------------------------------------------------------------------------------------

+# See also param-precommand-option-argument1.zsh
 alias sudo_u='sudo -u'
 sudo(){}

--- a/highlighters/main/test-data/alias-precommand-option-argument3.zsh
+++ b/highlighters/main/test-data/alias-precommand-option-argument3.zsh
@@ -28,6 +28,7 @@
 # vim: ft=zsh sw=2 ts=2 et
 # -------------------------------------------------------------------------------------------------

+# See also param-precommand-option-argument3.zsh
 alias sudo_u='sudo -u'
 sudo(){}

--- a/highlighters/main/test-data/alias-to-dir.zsh
+++ b/highlighters/main/test-data/alias-to-dir.zsh
@@ -32,5 +32,5 @@ alias x=/
 BUFFER=$'x'

 expected_region_highlight=(
-  '1 1 unknown-token "issue #202"' # x (/)
+  '1 1 unknown-token "issue #668"' # x (/)
 )
--- a/highlighters/main/test-data/assign-value-quote1.zsh
+++ b/highlighters/main/test-data/assign-value-quote1.zsh
@@ -0,0 +1,37 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'s="foo\'bar"'
+
+expected_region_highlight=(
+  '1 11 assign' # s="foo'bar"
+  '3 11 default' # "foo'bar"
+  '3 11 double-quoted-argument' # "foo'bar"
+)
--- a/highlighters/main/test-data/assign-value-quote2.zsh
+++ b/highlighters/main/test-data/assign-value-quote2.zsh
@@ -0,0 +1,37 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'s="foo \'\' bar"'
+
+expected_region_highlight=(
+  '1 14 assign' # s="foo '' bar"
+  '3 14 default' # "foo '' bar"
+  '3 14 double-quoted-argument' # "foo '' bar"
+)
--- a/highlighters/main/test-data/assignment-quoted.zsh
+++ b/highlighters/main/test-data/assignment-quoted.zsh
@@ -0,0 +1,37 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'1="foo"'
+
+expected_region_highlight=(
+  '1 7 assign' # 1="foo"
+  '3 7 default' # "foo"
+  '3 7 double-quoted-argument' # "foo"
+)
--- a/highlighters/main/test-data/backslash.zsh
+++ b/highlighters/main/test-data/backslash.zsh
@@ -0,0 +1,34 @@
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2018 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'\\'
+
+expected_region_highlight=(
+  '1 1 unknown-token' # \\
+)
--- a/highlighters/main/test-data/cmdpos-elision-partial.zsh
+++ b/highlighters/main/test-data/cmdpos-elision-partial.zsh
@@ -0,0 +1,44 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+# Test elision of some, but not all of the words
+# See issue #667 for the case of eliding all words
+local -a x; x=(sudo "")
+
+sudo(){}
+BUFFER=$'$x -u phy1729 ls'
+
+expected_region_highlight=(
+  '1 2 precommand' # $x
+  # The "" is elided.  If it weren't elided, the «ls» would be highlighted as an ordinary argument.
+  '4 5 single-hyphen-option' # -u
+  '7 13 default' # phy1729
+  '15 16 command' # ls
+)
--- a/highlighters/main/test-data/null-exec.zsh
+++ b/highlighters/main/test-data/null-exec.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'exec >/dev/null;'
+
+expected_region_highlight=(
+  '1 4 precommand' # exec
+  '6 6 redirection' # >
+  '7 15 path' # /dev/null
+  '16 16 commandseparator "issue #676"' # ;
+)
--- a/highlighters/main/test-data/param-precommand-option-argument1.zsh
+++ b/highlighters/main/test-data/param-precommand-option-argument1.zsh
@@ -0,0 +1,42 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+# See also alias-precommand-option-argument1.zsh
+local -a sudo_u; sudo_u=(sudo -u)
+sudo(){}
+
+BUFFER='$sudo_u phy1729 echo foo'
+
+expected_region_highlight=(
+  '1 7 precommand' # $sudo_u
+  '9 15 default' # phy1729
+  '18 20 command "issue #540"' # echo (not builtin)
+  '22 24 default' # foo
+)
--- a/highlighters/main/test-data/param-precommand-option-argument3.zsh
+++ b/highlighters/main/test-data/param-precommand-option-argument3.zsh
@@ -0,0 +1,42 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+# See also alias-precommand-option-argument3.zsh
+local -a sudo_u; sudo_u=(sudo -u)
+sudo(){}
+
+BUFFER='$sudo_u phy1729 ls foo'
+
+expected_region_highlight=(
+  '1 7 precommand' # sudo_u
+  '9 15 default' # phy1729
+  '17 18 command' # ls
+  '20 22 default' # foo
+)
--- a/highlighters/main/test-data/parameter-expansion-untokenized1.zsh
+++ b/highlighters/main/test-data/parameter-expansion-untokenized1.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local x="()"
+
+BUFFER=$'$x ls'
+
+expected_region_highlight=(
+  '1 2 unknown-token' # $x
+  '4 5 command' # ls
+)
--- a/highlighters/main/test-data/parameter-expansion-untokenized2.zsh
+++ b/highlighters/main/test-data/parameter-expansion-untokenized2.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local x="^foo^bar"
+
+BUFFER=$'$x ls'
+
+expected_region_highlight=(
+  '1 2 unknown-token' # $x
+  '4 5 default' # ls
+)
--- a/highlighters/main/test-data/parameter-value-contains-command-position1.zsh
+++ b/highlighters/main/test-data/parameter-value-contains-command-position1.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local foobar='x=$(ls)'
+
+BUFFER=$'$foobar'
+
+expected_region_highlight=(
+  # Used to highlight the "ba" as 'command' because the 'ls' showed through; issues #670 and #674
+  '1 7 unknown-token' # $foobar (not an assignment)
+)
--- a/highlighters/main/test-data/parameter-value-contains-command-position2.zsh
+++ b/highlighters/main/test-data/parameter-value-contains-command-position2.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+local y='x=$(ls)'
+
+BUFFER=$'$y'
+
+expected_region_highlight=(
+  # Used to trigger a "BUG" message on stderr - issues #670 and #674
+  '1 2 unknown-token' # $y (not an assignment)
+)
--- a/highlighters/main/test-data/pasted-quotes.zsh
+++ b/highlighters/main/test-data/pasted-quotes.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2013 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$': \'foo\'bar"baz"'
+
+expected_region_highlight=(
+  '1 1 builtin' # :
+  '3 15 default' # \'foo\'bar"baz"
+  '3 7 single-quoted-argument' # \'foo\'
+  '11 15 double-quoted-argument' # "baz"
+)
--- a/highlighters/main/test-data/precommand-killing1.zsh
+++ b/highlighters/main/test-data/precommand-killing1.zsh
@@ -0,0 +1,39 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+hash sudo=false
+
+BUFFER='sudo -e /etc/passwd'
+
+expected_region_highlight=(
+  '1 4 precommand' # sudo
+  '6 7 single-hyphen-option' # -e
+  '9 19 path' # /etc/passwd
+)
--- a/highlighters/main/test-data/precommand-killing2.zsh
+++ b/highlighters/main/test-data/precommand-killing2.zsh
@@ -0,0 +1,39 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+hash sudo=false
+
+BUFFER='sudo -e /does/not/exist'
+
+expected_region_highlight=(
+  '1 4 precommand' # sudo
+  '6 7 single-hyphen-option' # -e
+  '9 23 normal "issue #678"' # /does/not/exist
+)
--- a/highlighters/main/test-data/precommand-unknown-option.zsh
+++ b/highlighters/main/test-data/precommand-unknown-option.zsh
@@ -38,6 +38,6 @@ expected_region_highlight=(
  '10 11 command' # ls
  '12 12 commandseparator' # ;
  '14 17 precommand' # sudo
-  '19 20 unknown-token' # -x
+  '19 20 single-hyphen-option' # -x
  '22 23 command' # ls
 )
--- a/highlighters/main/test-data/process-substitution-after-redirection.zsh
+++ b/highlighters/main/test-data/process-substitution-after-redirection.zsh
@@ -0,0 +1,46 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2017, 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'< <(pwd) > >(nl)'
+
+expected_region_highlight=(
+  '1 1 redirection' # <
+  '3 8 default' # <(pwd)
+  '3 8 process-substitution' # <(pwd)
+  '3 4 process-substitution-delimiter' # <(
+  '5 7 builtin' # pwd
+  '8 8 process-substitution-delimiter' # )
+  '10 10 redirection' # >
+  '12 16 default' # >(nl)
+  '12 16 process-substitution' # >(nl)
+  '12 13 process-substitution-delimiter' # >(
+  '14 15 command' # nl
+  '16 16 process-substitution-delimiter' # )
+)
--- a/highlighters/main/test-data/quoted-redirection-in-command-word.zsh
+++ b/highlighters/main/test-data/quoted-redirection-in-command-word.zsh
@@ -0,0 +1,37 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'">" foo ls'
+
+expected_region_highlight=(
+  '1 3 unknown-token' # ">" - not "redirection"
+  '5 7 default' # foo
+  '9 10 default' # ls - not "command"
+)
--- a/highlighters/main/test-data/redirection-is-not-option.zsh
+++ b/highlighters/main/test-data/redirection-is-not-option.zsh
@@ -0,0 +1,38 @@
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2018 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$': > -x >> --yy'
+
+expected_region_highlight=(
+  '1 1 builtin' # :
+  '3 3 redirection' # >
+  '5 6 default' # -x
+  '8 9 redirection' # >>
+  '11 14 default' # --yy
+)
--- a/highlighters/main/test-data/sudo-longopt.zsh
+++ b/highlighters/main/test-data/sudo-longopt.zsh
@@ -0,0 +1,38 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+hash sudo='false'
+BUFFER='sudo --askpass ls'
+
+expected_region_highlight=(
+  '1 4 precommand' # sudo
+  '6 14 double-hyphen-option' # --askpass
+  '16 17 command' # ls (we don't know whether --askpass takes an argument)
+)
--- a/highlighters/main/test-data/time-and-nocorrect1.zsh
+++ b/highlighters/main/test-data/time-and-nocorrect1.zsh
@@ -0,0 +1,39 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2019 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'time ls; nocorrect ls'
+
+expected_region_highlight=(
+  '1 4 reserved-word' # time
+  '6 7 command' # ls
+  '8 8 commandseparator' # ;
+  '10 18 reserved-word' # nocorrect
+  '20 21 command' # ls
+)
--- a/highlighters/main/test-data/time-and-nocorrect2.zsh
+++ b/highlighters/main/test-data/time-and-nocorrect2.zsh
@@ -0,0 +1,40 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2019 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+BUFFER=$'time ls; nocorrect ls'
+alias time=':' nocorrect=':'
+
+expected_region_highlight=(
+  '1 4 alias' # time
+  '6 7 default' # ls
+  '8 8 commandseparator' # ;
+  '10 18 alias' # nocorrect
+  '20 21 default' # ls
+)
--- a/release.md
+++ b/release.md
@@ -5,11 +5,12 @@
 - Check open issues and outstanding pull requests
 - Confirm `make test` passes
  - check with multiple zsh versions
+    (easiest to check travis: https://travis-ci.org/zsh-users/zsh-syntax-highlighting/)
 - Update changelog.md
  `tig --abbrev=12  --abbrev-commit 0.4.1..upstream/master`
 - Remove `-dev` suffix from `./.version`;
  Commit that using `git commit -m "Tag version $(<.version)." .version`;
-  Tag it using `git tag -s -m "Tag version $(<.version)"`;
+  Tag it using `git tag -s -m "Tag version $(<.version)" $(<.version)`;
  Increment `./.version` and restore the `-dev` suffix;
  Commit that using `git commit -C b5c30ae52638e81a38fe5329081c5613d7bd6ca5 .version`.
 - Push with `git push && git push --tags`
--- a/tests/edit-failed-tests
+++ b/tests/edit-failed-tests
@@ -0,0 +1,40 @@
+#!/usr/bin/env zsh
+# -------------------------------------------------------------------------------------------------
+# Copyright (c) 2020 zsh-syntax-highlighting contributors
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification, are permitted
+# provided that the following conditions are met:
+#
+#  * Redistributions of source code must retain the above copyright notice, this list of conditions
+#    and the following disclaimer.
+#  * Redistributions in binary form must reproduce the above copyright notice, this list of
+#    conditions and the following disclaimer in the documentation and/or other materials provided
+#    with the distribution.
+#  * Neither the name of the zsh-syntax-highlighting contributors nor the names of its contributors
+#    may be used to endorse or promote products derived from this software without specific prior
+#    written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR
+# IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND
+# FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+# IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# -------------------------------------------------------------------------------------------------
+# -*- mode: zsh; sh-indentation: 2; indent-tabs-mode: nil; sh-basic-offset: 2; -*-
+# vim: ft=zsh sw=2 ts=2 et
+# -------------------------------------------------------------------------------------------------
+
+type perl sponge >/dev/null || { print -ru2 -- "$0: This script requires perl(1) and sponge(1) [from moreutils]"; exit 1; }
+
+local editor=( "${(@Q)${(z)${VISUAL:-${EDITOR:-vi}}}}" )
+() { 
+  > "$2" perl -nE '$highlighter = $1 if /^Running test (\S*)/; say "highlighters/${highlighter}/test-data/$1.zsh" if /^# (\S*)/' "$1"
+  >>"$2" echo ""
+  >>"$2" cat <"$1"
+  "${editor[@]}" -- "$2"
+} =(${MAKE:-make} quiet-test) =(:)
+# TODO: tee(1) the quiet-test output to /dev/tty as it's happening, with colors.
--- a/tests/generate.zsh
+++ b/tests/generate.zsh
@@ -32,16 +32,17 @@ emulate -LR zsh
 setopt localoptions extendedglob

 # Argument parsing.
-if (( $# != 3 )) || [[ $1 == -* ]]; then
-  print -r -- >&2 "$0: usage: $0 BUFFER HIGHLIGHTER BASENAME"
+if (( $# * $# - 7 * $# + 12 )) || [[ $1 == -* ]]; then
+  print -r -- >&2 "$0: usage: $0 BUFFER HIGHLIGHTER BASENAME [PREAMBLE]"
  print -r -- >&2 ""
  print -r -- >&2 "Generate highlighters/HIGHLIGHTER/test-data/BASENAME.zsh based on the"
-  print -r -- >&2 "current highlighting of BUFFER."
+  print -r -- >&2 "current highlighting of BUFFER, using the setup code PREAMBLE."
  exit 1
 fi
 buffer=$1
 ZSH_HIGHLIGHT_HIGHLIGHTERS=( $2 )
 fname=${0:A:h:h}/highlighters/$2/test-data/${3%.zsh}.zsh
+preamble=${4:-""}

 # Load the main script.
 . ${0:A:h:h}/zsh-syntax-highlighting.zsh
@@ -55,13 +56,18 @@ _zsh_highlight_add_highlight()

 # Copyright block
 year="`LC_ALL=C date +%Y`"
-if ! read -q "?Set copyright year to $year? "; then
+if ! { read -q "?Set copyright year to $year? " } always { echo "" }; then
  year="YYYY"
 fi
-exec >$fname
-<$0 sed -n -e '1,/^$/p' | sed -e "s/2[0-9][0-9][0-9]/${year}/"
+<$0 sed -n -e '1,/^$/p' | sed -e "s/2[0-9][0-9][0-9]/${year}/" > $fname
 # Assumes stdout is line-buffered
 git add -- $fname
+exec > >(tee -a $fname)
+
+# Preamble
+if [[ -n $preamble ]]; then
+  print -rl -- "$preamble" ""
+fi

 # Buffer
 print -n 'BUFFER='
@@ -82,8 +88,17 @@ print 'expected_region_highlight=('
  PREBUFFER=""
  BUFFER="$buffer"
  region_highlight=()
-  # TODO: use run_test() from tests/test-highlighting.zsh (to get a tempdir)
-  _zsh_highlight
+  eval $(
+    exec 3>&1 >/dev/null
+    typeset -r __tests_tmpdir="$(mktemp -d)"
+    {
+      # Use a subshell to ensure $__tests_tmpdir, which is to be rm -rf'd, won't be modified.
+      (cd -- "$__tests_tmpdir" && eval $preamble && _zsh_highlight && typeset -p region_highlight >&3)
+      : # workaround zsh bug workers/45305 with respect to the $(…) subshell we're in
+    } always {
+      rm -rf -- ${__tests_tmpdir}
+    }
+  )

  for ((i=1; i<=${#region_highlight}; i++)); do
    local -a highlight_zone; highlight_zone=( ${(z)region_highlight[$i]} )
--- a/tests/tap-filter
+++ b/tests/tap-filter
@@ -40,6 +40,8 @@ use strict;

 undef $/; # slurp mode
 print for
-  grep { /^ok.*# TODO/m or /^not ok(?!.*# TODO)/m }
-    split /^(?=#)/m,
+  grep { /^ok.*# TODO/m or /^not ok(?!.*# TODO)/m or /^Bail out!/m }
+    # Split on plan lines and remove them from the output.  (To keep them,
+    # use the lookahead syntax, «(?=…)», to make the match zero-length.)
+    split /^\d+\.\.\d+$/m,
      <STDIN>;
--- a/tests/test-highlighting.zsh
+++ b/tests/test-highlighting.zsh
@@ -94,7 +94,11 @@ ZSH_HIGHLIGHT_HIGHLIGHTERS=($1)
 # In zsh<5.3, 'typeset -p arrayvar' emits two lines, so we use this wrapper instead.
 typeset_p() {
  for 1 ; do
+    if [[ ${(tP)1} == *array* ]]; then
      print -r -- "$1=( ${(@q-P)1} )"
+    else
+      print -r -- "$1=${(q-P)1}"
+    fi
  done
 }

@@ -113,12 +117,11 @@ run_test_internal() {
  local srcdir="$PWD"
  builtin cd -q -- "$tests_tempdir" || { echo >&2 "Bail out! On ${(qq)1}: cd failed: $?"; return 1 }

-  echo "# ${1:t:r}"
-
  # Load the data and prepare checking it.
  local BUFFER CURSOR MARK PENDING PREBUFFER REGION_ACTIVE WIDGET REPLY skip_test unsorted=0
  local expected_mismatch
  local -a expected_region_highlight region_highlight
+
  . "$srcdir"/"$1"

  (( $#skip_test )) && { print -r -- "1..0 # SKIP $skip_test"; return; }
@@ -139,7 +142,12 @@ run_test_internal() {
    expected_region_highlight=("${(@n)expected_region_highlight}")
  fi

+  # Print the plan line, and some comments for human readers
  echo "1..$(( $#expected_region_highlight + 1))"
+  echo "## ${1:t:r}"
+  [[ -n $PREBUFFER ]] && printf '# %s\n' "$(typeset_p PREBUFFER)"
+  [[ -n $BUFFER ]] && printf '# %s\n' "$(typeset_p BUFFER)"
+
  local i
  for ((i=1; i<=$#expected_region_highlight; i++)); do
    local -a expected_highlight_zone; expected_highlight_zone=( ${(z)expected_region_highlight[i]} )
Author	SHA1	Message	Date
Daniel Shahaf	36a3c0f82c	Tag version 0.7.0.	2020-02-28 15:36:43 +00:00
Daniel Shahaf	d65f4f8a35	release.md: Add details about a step.	2020-02-28 15:36:25 +00:00
Daniel Shahaf	e07c901dfd	tests: Fix the last added test to pass when sudo(8) isn't installed.	2020-02-28 15:22:43 +00:00
Daniel Shahaf	3cea1434ae	Bump copyright years.	2020-02-25 17:34:35 +00:00
Daniel Shahaf	73c89c69a2	tests: Add a test for partial elisions of parameter expansions in command position See `1a55dc8fc2 (commitcomment-37476021)`	2020-02-25 17:22:55 +00:00
Matthew Martin	7fd44bc429	tests: Fix previous	2020-02-25 07:37:09 -06:00
Matthew Martin	ab88dfad27	tests: Add main test for alias of a parameter like string	2020-02-25 07:21:40 -06:00
Daniel Shahaf	2b3638a211	test harness: Tweak quiet-test output - Print the test name and data after the plan line - Split on the plan line rather than on comments + That makes tap-filter more suitable to filter TAP output generated by other TAP producers. + However, the filtered output deletes the plan line and adds a blank line in its stead. This suits our use-case of interactive test runs.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	3414c7c0d2	test harness: Include $PREBUFFER and $BUFFER in the output. For human readers' benefit.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	4a043b4d15	noop: Whitespace changes only. ./.editorconfig is already set correctly.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	dc70e89bfd	tests: Support non-arrays in typeset_p().	2020-02-22 15:12:24 +00:00
Daniel Shahaf	f729726300	'main': Do not look for metacharacters in parameter expansions. Fixes the bug the previous commit added a test for.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	f490b7cb95	'main': Add two tests for metacharacters in parameter expansions. Suggested-by: @QBobWatson (in https://github.com/zsh-users/zsh-syntax-highlighting/pull/682#issuecomment-588361771)	2020-02-22 15:12:24 +00:00
Daniel Shahaf	2f4f81cab7	'main': Parameter expansions may not contain assignments. In «a="b=c"; $a», the '=' sign in the expansion of $a is not active. Therefore, prevent the expansion of $a from being considered an assignment. Update test expectations accordingly.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	3558306149	tests: Add tests for issue #670 . Before the parent commit, they behaved as follows: ZSH_PATCHLEVEL=debian/5.7.1-1 # parameter-value-contains-command-position1 1..2 ok 1 - [1,7] «$foobar» - # TODO "issue #670" not ok 2 - have 1 expectations and 6 region_highlight entries: «expected_region_highlight=( '1 7 assign "issue ♯670"' )» «region_highlight=( '0 7 assign' '2 7 default' '2 7 command-substitution-unquoted' zsh-syntax-highlighting: BUG: _zsh_highlight_highlighter_main_paint: start(2) >= end(2) Bail out! On './highlighters/main/test-data/parameter-value-contains-command-position2.zsh': output on stderr # parameter-value-contains-command-position2 1..2 ok 1 - [1,2] «$y» - # TODO "issue #670" ok 2 - cardinality check Due to the the "BUG:" and "Bail out!" on the first one, they could not be added as XFAIL tests before the parent commit.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	1a55dc8fc2	'main': Pass parameters through the multi-word machinery, as we already do for aliases. Fixes #674.	2020-02-22 15:12:24 +00:00
Daniel Shahaf	52ea5c686a	'main': precommands += chronic, ifne (from moreutils) Fixes #681.	2020-02-17 10:20:13 +00:00
Manaswini Das	dde84e1b25	docs: Fix typo Fixes #679	2020-01-28 07:44:52 +00:00
Matthew Martin	0e51046b19	main: Add tests for issue #678	2020-01-25 15:51:16 -06:00
Daniel Shahaf	4e92449752	changelog: Document #670 as a known issue.	2020-01-25 21:00:40 +00:00
Daniel Shahaf	b73853dd96	changelog: Document #677 as a known issue.	2020-01-24 18:40:51 +00:00
Daniel Shahaf	4546756500	tests: Fix another instance of issue from the last commit.	2020-01-24 01:22:03 +00:00
Daniel Shahaf	9d380805d7	tests: Unbreak the build on zsh 5.0.8 and older.	2020-01-24 00:51:27 +00:00
Daniel Shahaf	deee22ed42	tests: Don't filter out tests that aborted. Useful in piping Travis CI output through tap-filter manually.	2020-01-24 00:49:51 +00:00
Daniel Shahaf	77c6bf2019	'main': precommands: Remove argumentless options that can't be followed by a command word. Suggested-by: Matthew Martin (in `e2e97dde9c (r36941988)`)	2020-01-24 00:37:56 +00:00
Daniel Shahaf	27e4789439	docs: Track FreeBSD port rename See https://svnweb.freebsd.org/ports?revision=501751&view=revision	2020-01-23 22:09:33 +00:00
Daniel Shahaf	e2e97dde9c	'main': Support tabbed(1) from suckless-tools	2020-01-22 04:44:27 +00:00
Daniel Shahaf	9bf06c5c2a	tests: Add a regression test for issue #676 .	2020-01-20 03:49:00 +00:00
Daniel Shahaf	dfb917020c	Bump copyright years.	2020-01-16 17:19:25 +00:00
Daniel Shahaf	9ed2a46ed2	dev tools: New script to ease opening $EDITOR on failing tests. Has room for improvement; for now, I use it with CTRL-W_f.	2020-01-16 17:19:25 +00:00
Daniel Shahaf	1c6a6d92b0	dev tools: Allow specifying preamble code when generating test cases.	2020-01-16 17:19:25 +00:00
Daniel Shahaf	6b0671b63e	Add .editorconfig file. Vim doesn't honour the modelines by default because they're too far from the start/end of the file [the default value of the 'modelines' (sic) option is too small].	2020-01-16 17:19:25 +00:00
Daniel Shahaf	b3b6d7129f	tests: Add failing tests for issue #674 .	2020-01-16 16:12:14 +00:00
Daniel Shahaf	9cb7e9a837	tests: Make sudo-longopt pass when sudo isn't installed. Fixes #673.	2020-01-16 15:49:27 +00:00
Daniel Shahaf	cb166dcc77	'main': Restore 0.6.0's behaviour for unknown options. See discussion on #664.	2020-01-14 23:08:50 +00:00
Daniel Shahaf	5012d771f3	'main': Don't highlight unknown precommand flags as errors. We cannot now for sure whether the flag is misspelled or simply unknown to us, so err on the side of caution. This fixes an unreleased regression. Fixes #658. Issue #641 was originally filed about this problem, but is left open to track further enhancements.	2020-01-14 23:08:50 +00:00
Daniel Shahaf	3a6f7e7bfe	'main': Add an XFail test for #641 and #658 .	2020-01-14 23:08:50 +00:00
Daniel Shahaf	521fedfdf5	'main': Change the issue a test is associated with. Issue #202 is too overloaded.	2020-01-12 19:32:44 +00:00
Daniel Shahaf	1618848df6	'main': Add some tests for complete and partial absolute paths in command position.	2020-01-12 19:29:46 +00:00
Daniel Shahaf	c3293ba0d8	'main': Update comments after last commit. No functional change.	2020-01-12 18:34:49 +00:00
Daniel Shahaf	21cdd6bc5e	'main': Simplify alias handling. $last_alias isn't needed; there's no reason to treat loops of length 2 (alias a=b b=a) differently to loops of length 1 (alias a=a), length 3 (alias a=b b=c c=a), or length N. The «(( $+seen_alias[$arg] ))» check is redundant as of the last commit: the enclosing condition ensures that $res is "alias", which implies that «(( $+seen_alias[$arg] ))» is false.	2020-01-12 18:34:49 +00:00
Daniel Shahaf	f32d1704b1	'main': Fix issue #652 .	2020-01-12 18:34:49 +00:00
Daniel Shahaf	86fe054005	'main': precommands += ssh-agent	2020-01-12 17:21:59 +00:00
Daniel Shahaf	533bfa0116	'main': Fix the new 'backslash' test on zsh-5.0.2 and older. (The bug occurred on zsh-5.0.7 and older but I don't have zsh-5.0.7 handy to test on.) Evidently, the issue was due to elision. This addresses #665.0 and #665.5.	2020-01-12 17:10:34 +00:00
Daniel Shahaf	2d0dddf58b	'main': Don't dequote the word in command position before analyzing it. Fixes #656. Fixes #660. Regression from commit `e1ecf950e2`, "main: Do path expanstion after alias expansion" (sic).	2020-01-12 16:11:09 +00:00
Daniel Shahaf	27fa4a6546	noop: Quote the bitflag values. Now every instance of «:foo:» is ''-quoted. This enables $EDITOR to highlight them consistently throughout the file.	2020-01-12 15:46:22 +00:00
Daniel Shahaf	ae5ad09cf5	Bump copyright years.	2020-01-12 14:58:56 +00:00
Daniel Shahaf	3ee5aa0e06	release.md: Add missing argument.	2020-01-11 23:43:23 +00:00
Daniel Shahaf	eb277cdcc6	Post-release version number bump.	2020-01-11 23:42:31 +00:00
Daniel Shahaf	71dc090d02	Tag version 0.7.0-beta1.	2020-01-11 23:34:58 +00:00
Daniel Shahaf	6a310ada5b	changelog: Add three entries	2020-01-11 23:31:18 +00:00
Daniel Shahaf	c1d5790c57	changelog: Start 0.7.0's changelog.	2020-01-11 23:10:28 +00:00
Daniel Shahaf	e9b1ce1a36	Add a unit test for `dc1b2f6fa4`.	2020-01-11 23:04:22 +00:00
Daniel Shahaf	68fbe1a449	Add a test for redirection from/to process substitution. Inspired by `ab1013ae0d`.	2020-01-11 21:47:34 +00:00
Daniel Shahaf	f02c0bf980	Don't describe as ZSH_HIGHLIGHT_DIRS_BLACKLIST as experimental, now that it's named without the X_ prefix.	2020-01-11 21:32:21 +00:00
Daniel Shahaf	1be58a6f45	Add a test for `2f03b6d704`.	2020-01-11 21:19:52 +00:00
Daniel Shahaf	c0ad50e645	dev tools: Fix regression introduced in commit "Print the test data to stdout for convenience.". tee(1) truncated $fname. Fortunately, the data that got truncated had just been `git add`-ed, so no harm was done.	2020-01-11 20:49:12 +00:00
Daniel Shahaf	8b2768ab40	'main': Add a test for `07f259f653`.	2020-01-11 20:46:14 +00:00
Daniel Shahaf	6f1f595e3e	dev tools: Print the test data to stdout for convenience.	2020-01-07 19:59:49 +00:00
Daniel Shahaf	de95d50bce	'main': Fix highlighting of the 'time' and 'nocorrect reserved words Also add tests. 'time' is a regression from `6647e88606` (last Wednesday).	2019-12-27 13:47:31 +00:00
Daniel Shahaf	83862c1abb	'main': Add two basic tests for aliases.	2019-12-27 09:37:23 +00:00