Gmod Lua Lexer: A Beginner’s Guide to Tokenizing Garry’s Mod Scripts

1. Initialize position, line, col, state = Default.
2. While not end:
   • If state == Default:
     • Skip whitespace; update pos/line/col.
     • If next two chars == ‘–’ then enter Comment (line or long).
     • If char == ‘“’ or “‘” enter String, record delimiter.
     • If char == ‘[’ check for long bracket; if so enter LongBracket.
     • Match identifiers/keywords via pattern; numbers via numeric pattern.
     • Emit token for operators/punctuation (handle two-char operators like ==, <=, =).
   • If state == String:
     • Consume until unescaped delimiter; handle escapes; emit STRING token.
   • If state == LongBracket:
     • Scan until matching closing bracket level; emit LONG_STRING or LONG_COMMENT.
   • If state == Comment:
     • Consume to end-of-line; emit COMMENT token.

Handling edge cases

• Unterminated strings/long brackets: lexers should report clear diagnostics with line/col and attempt to recover (e.g., treat rest of file as string or stop at EOF).
• Nested long brackets: Lua does not nest by delimiter; treat inner brackets as content.
• Escape sequences: decide whether to unescape string values in lexer or leave raw text for parser.
• CRLF vs LF: normalize newlines consistently for line/column tracking.
• Performance: avoid repeated substring allocations. Use indices into the original source and only allocate token.value when needed.
• Large files / addons: stream processing or chunked reading reduces memory usage.

Gmod-specific considerations

• Many addons include data files or chunked code in comments — consider scanning comments for annotations (@param, @server) and emitting structured annotation tokens.
• Common patterns like if SERVER then/if CLIENT then might be used by tools to split files into client/server parts. A post-lexing pass that recognizes these conditional blocks can be practical.
• Sandboxed environments and custom preprocessors: if your tool must operate on packed/obfuscated code, add a preprocessing stage (decompression, deobfuscation) before lexing.

Testing and validation

• Create fuzz tests with random inputs including all edge constructs (unterminated strings, long brackets with varying = counts, odd Unicode identifiers).
• Unit tests for token sequences for representative Gmod addons: weapons, gamemodes, HUDs.
• Performance benchmarks on large addons and shared repositories.

Sample minimal Lua lexer snippet (conceptual)

lua
Copy CodeCopied
– conceptual: not production-ready
local function lex(source)
  local pos, len, line, col = 1, #source, 1, 1
  local tokens = {}
  local function emit(type, value)
    tokens[#tokens+1] = {type=type, value=value, line=line, col=col}
  end
  while pos <= len do
    local ch = source:sub(pos,pos)
    if ch:match(”%s”) then
      if ch == ” “ then line = line + 1; col = 1 else col = col + 1 end
      pos = pos + 1
    elseif ch == ”-” and source:sub(pos,pos+1) == ”–” then
      – line comment
      local s,e = source:find(” “, pos+2) or (len+1)
      emit(“COMMENT”, source:sub(pos, e-1))
      pos = e     elseif ch == ’“’ or ch == ”‘” then
      local start = pos       pos = pos + 1
      while pos <= len do
        local c = source:sub(pos,pos)
        if c == ”\” then pos = pos + 2
        elseif c == ch then pos = pos + 1; break
        else pos = pos + 1 end
      end
      emit(“STRING”, source:sub(start, pos-1))
    else
      – identifiers, numbers, operators simplified…
      pos = pos + 1
    end
  end
  return tokens end

Conclusion

A robust Gmod Lua lexer balances correctness (handling Lua’s nuanced long brackets and escape rules), performance (minimal copying, streaming where needed), and Gmod-specific duties (annotations, server/client splitting). Implement explicit lexer states, precise long-bracket matching, and thorough tests. For tooling, consider a post-lexing pass to extract Gmod annotations and conditional blocks.