Add tokens and lexer! by ujaandas · Pull Request #10 · ujaandas/invariants

ujaandas · 2026-04-23T22:08:12Z

This PR adds the first-steps and initial foundation for the invariants lang, the lexer! The lexer is responsible for converting raw invariants code into a series of tokens.

The set of tokens consists of all possible symbols, literals and otherwise keywords of importance, such as spec, ==, +, Boolean, etc... A literal, in this context, is the "literal" value associated with a token, and is either a std::monostate, for tokens with no literal values (eg; operators, keywords, punctuation, nulls), a std::string for identifiers and string-type variables, int for integers, double for "numbers" (as per the OpenAPI spec), or bool for booleans. Some helper operators such as ostream<< and equality checks have also been implemented. For the most part, this is just a fairly straightforward representation of data.

The lexer class is more interesting, as it parses our text and converts it into a stream of tokens. The implementation thereof is fairly straightforward, as thanks to our token struct inheriting from uint8, we can use a switch/case table which devolves into a jump-table post-compilation for what I imagine would be a significant performance boost as opposed to using a map to match characters to tokens. We, at most, lookahead 2 characters.

A very basic supported (scannable) input might look like this:

spec User {
   field age: Integer
   check age >= 18
}

Also, a very, very, very basic interpreter has been added. Some more work needs to be done surrounding the actual CLI and "framework" of the language, such as adding a REPL, proper error types, etc... but that is out of scope of this branch and can be tackled much later as those are more niceties than anything.

Also, we make extensive use of GTest and (ideally) have tested most common paths.

* do not pass filenames to cppcheck, just use compile db * add nix run configure just in case

* add token equality operator * add ctor for optional literal * rename tests for parity * test on tokens directly, not string repr

Copilot

Pull request overview

This PR introduces the initial lexer/token infrastructure (plus a minimal interpreter) for the invariants language, along with a Nix/prek workflow to build and run tests.

Changes:

Added TokenType/Token model with printing, equality, and literal support.
Implemented a Lexer that scans source text into a token stream and added GTest coverage for tokens/lexing.
Added a minimal Interpreter that runs the lexer and prints tokens; wired new libs/tests into CMake and added nix run .#test.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
prek.toml	Switches to local prek hooks and adds Nix-based configure/format/tidy/cppcheck/test hooks.
flake.nix	Adds `dev-test` app and exposes `nix run .#test`.
README.md	Documents running the test suite via Nix app.
lang/src/lexer/token.hpp	Adds token types, literal variant, and token API.
lang/src/lexer/token.cpp	Implements token formatting and comparison.
lang/src/lexer/lexer.hpp	Declares lexer and keyword table.
lang/src/lexer/lexer.cpp	Implements scanning logic for tokens, literals, comments, whitespace.
lang/src/interp/interpreter.hpp	Declares minimal interpreter skeleton.
lang/src/interp/interpreter.cpp	Implements `run()` by lexing and printing tokens.
lang/src/**/CMakeLists.txt	Builds lexer/interpreter libs and wires them into the main executable.
lang/tests/**	Adds GTest suites for token/lexer and a basic interpreter test; updates test CMake structure.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+#include "token.hpp"
+
+#include <ostream>
+#include <string>


+#pragma once
+
+#include <string>
+#include <unordered_map>
+#include <vector>


+          addToken(type, false);
+          return;
+        }
+
+        addToken(type);


+#include <cstddef>
+#include <string_view>
+
+namespace invariants::interpreter {
+
+class Interpreter {
+ private:
+  bool hadErr = false;
+  void report(std::size_t line, std::string_view where, std::string_view msg);
+


+// #include <string_view>
+
+#include "lexer.hpp"
+
+namespace invariants::interpreter {
+
+// void Interpreter::report(std::size_t line, std::string_view where,
+//                          std::string_view msg) {
+//   std::println("[line %d] Error %s : %s", line, where, msg);
+//   hadErr = true;
+// }


+  return !(this->type == other.type && this->lexeme == other.lexeme &&
+           this->literal == other.literal && this->line == other.line);


+  bool operator!=(const Token& other) const;
+  std::string toString() const;
+
+  friend std::ostream& operator<<(std::ostream& os, const Token& token);
+};


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

This PR introduces the initial C++ implementation of the invariants language lexer/token model (plus a minimal interpreter stub) and wires them into the build/test + developer tooling.

Changes:

Add TokenType/Token (with literals, formatting, and equality) and a Lexer that scans source text into tokens.
Add initial Interpreter::run() that lexes input and prints the token stream.
Add GoogleTest coverage for token and lexer behavior; update CMake/Nix/prek tooling to build and run tests.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`prek.toml`	Switch hooks to local/system hooks invoking Nix commands (configure/format/tidy/cppcheck/test).
`lang/src/lexer/token.hpp`	Introduces token types, literal representation, and `Token` API.
`lang/src/lexer/token.cpp`	Implements token formatting and equality helpers.
`lang/src/lexer/lexer.hpp`	Declares lexer and keyword table.
`lang/src/lexer/lexer.cpp`	Implements scanning logic for punctuation/operators/literals/keywords/comments.
`lang/src/lexer/CMakeLists.txt`	Builds `invariants_lexer` library.
`lang/src/interp/interpreter.hpp` / `interpreter.cpp`	Adds minimal interpreter that runs the lexer and prints tokens.
`lang/src/interp/CMakeLists.txt`	Builds `invariants_interp` library.
`lang/src/CMakeLists.txt` / `lang/CMakeLists.txt`	Adds subdirectories and links `hello_world` to lexer.
`lang/tests/lexer/` + `lang/tests/interp/` + `lang/tests/CMakeLists.txt`	Adds/organizes lexer/token/interpreter tests and discovery.
`flake.nix`	Adds `nix run .#test` app and wires it into flake outputs.
`README.md`	Documents `nix run .#test` and notes about impure scripts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+std::string literalToString(const Literal& lit) {
+  return std::visit(
+      [](const auto& value) -> std::string {
+        using T = std::decay_t<decltype(value)>;
+
+        if constexpr (std::is_same_v<T, std::monostate>) {
+          return "null";
+        } else if constexpr (std::is_same_v<T, std::string>) {
+          return value;
+        } else if constexpr (std::is_same_v<T, double>) {
+          return std::to_string(value);
+        } else if constexpr (std::is_same_v<T, int>) {
+          return std::to_string(value);
+        } else if constexpr (std::is_same_v<T, bool>) {
+          return value ? "true" : "false";
+        }


+  const std::string source;
+  std::vector<Token> tokens;
+  inline static const std::unordered_map<std::string_view, TokenType> keywords{
+      {"spec", TokenType::KW_SPEC},
+      {"field", TokenType::KW_FIELD},
+      {"check", TokenType::KW_CHECK},
+      {"invariant", TokenType::KW_INVARIANT},
+      {"Boolean", TokenType::KW_BOOLEAN},
+      {"true", TokenType::LIT_BOOLEAN_T},
+      {"false", TokenType::LIT_BOOLEAN_F},
+      {"Array", TokenType::KW_ARRAY},
+      {"Null", TokenType::KW_NULL},
+      {"null", TokenType::LIT_NULL},
+      {"String", TokenType::KW_STRING},
+      {"Number", TokenType::KW_NUMBER},
+      {"Integer", TokenType::KW_INTEGER},
+      {"IN", TokenType::KW_IN},
+      {"NIN", TokenType::KW_NOT_IN},
+      {"NI", TokenType::KW_CONTAINS},
+  };
+  size_t start = 0;
+  size_t curr = 0;
+  size_t line = 1;
+
+  void scanToken();
+  char advance();
+  void addToken(TokenType type);
+  void addToken(TokenType type, Literal literal);
+
+ public:
+  explicit Lexer(std::string_view source);
+  std::vector<Token> scanTokens();


+        std::string text = source.substr(start, curr - start);
+
+        auto it = keywords.find(text);
+        TokenType type =
+            (it != keywords.end()) ? it->second : TokenType::LIT_IDENTIFIER;
+
+        // Check if boolean and if so, add relevant literals
+        if (type == TokenType::LIT_BOOLEAN_T) {
+          addToken(type, true);
+          return;
+        }
+
+        if (type == TokenType::LIT_BOOLEAN_F) {
+          addToken(type, false);
+          return;
+        }
+
+        addToken(type);
+      }


        dev-configure = pkgs.writeShellApplication {
          name = "dev-configure";
          meta.description = "Configure clangd environment.";
          runtimeInputs = with pkgs; [
            clang
            cmake
            ninja
          ];
          text = ''
            set -euo pipefail
            cmake -S lang -B .nix-dev/build
          '';


+        dev-test = pkgs.writeShellApplication {
+          name = "dev-test";
+          meta.description = "Run test suite.";
+          runtimeInputs = with pkgs; [
+            cmake
+            ninja
+          ];
+          text = ''
+            set -euo pipefail
+            cmake -S lang -B .nix-dev/build
+            cmake --build .nix-dev/build
+            ctest --test-dir .nix-dev/build --output-on-failure
+          '';


+
+  friend std::ostream& operator<<(std::ostream& os, const Token& token);
+};


+using Literal = std::variant<std::monostate,  // null
+                             std::string,     // identifiers + strings
+                             int,             // integers
+                             double,          // numbers
+                             bool             // booleans
+                             >;


ujaandas and others added 30 commits April 3, 2026 23:41

add initial tokentypes

d8f7a18

add cmakelists to lexer and tests

b2c1164

Chore/lang/cppcheck use cdb (#6)

b4f3d69

* do not pass filenames to cppcheck, just use compile db * add nix run configure just in case

move all hooks to local use nix develop (#7)

f32d4a8

extend token class, impl literal variant

c0c271f

test extended token class

8bca384

add test app

0218e6a

add tests to precommit

80fa682

move tests to pre push

6f54fb6

rename test

d71c73a

update readme to add nix run test doc

1f4bbae

wip lexer api

5fc4914

add tests for lexer api, make cppcheck changes

188859e

simplify transform in token stringify

19acae5

add lexer source wip

f706243

add cmake and support multichar tokens

75b96f8

test multichar tokens

20bcef4

formatting fix

2542619

add comment and fix slash support

39880a9

add comment and slash tests

cf87e4f

small tweaks, use advance() instead of curr++

357a68d

add lexer support for whitespace chars

3155dab

test some whitespaces

00808ed

Destringify lexer tests (#9)

5d14ff5

* add token equality operator * add ctor for optional literal * rename tests for parity * test on tokens directly, not string repr

add tests for multiline

38bb5f5

add string literal support and basic test

aee7f60

add number literal support

7cb29d0

split literal bool token into t/f

d11abe5

add keyword map

39a2dea

add keyword support

91fe37c

ujaandas added 14 commits April 23, 2026 17:44

allow underscores in identifier names

aa426a9

rename nil to null

fede4ca

add tests for single-char tokens

cee17a9

use parametrized tests for simple token checks

bc81713

add parametrized tests for structural keywords

87485ae

overload << operator for pretty printing in tests

fac0178

test type keywords too

3859e8c

add operator tests

c702ab0

rename test suite

3a51dd8

add some more token tests

cb49192

add a bunch more lexer tests

51de7b9

test interp too

80ef884

fix state reset on scanTokens

aba3809

more tests for lexer

bc525d2

Copilot AI review requested due to automatic review settings April 23, 2026 22:08

Copilot started reviewing on behalf of ujaandas April 23, 2026 22:08 View session

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Cast peeked char to unsigned char for std::isdigit

9186202

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings April 23, 2026 22:14

Copilot started reviewing on behalf of ujaandas April 23, 2026 22:14 View session

ujaandas added 2 commits April 23, 2026 22:18

dont rely on transitive hdrs

b29d668

dont rely on transitive hdrs, use std::size_t

f8aac8f

Copilot AI reviewed Apr 23, 2026

View reviewed changes

ujaandas added 5 commits April 23, 2026 22:22

give LIT_IDENTIFIER std::string literal vals

90d5696

clean up interpo

86907fd

fix neq operator to just negate existing eq op

d184cb8

fwd decl for op<< overload in token

756a4a5

clean up flake, add more deps just in case

98d201b

ujaandas merged commit 2d728ab into main Apr 23, 2026
1 check passed

ujaandas deleted the feat/lang/lexer branch April 23, 2026 22:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tokens and lexer!#10

Add tokens and lexer!#10
ujaandas merged 59 commits intomainfrom
feat/lang/lexer

ujaandas commented Apr 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return !(this->type == other.type && this->lexeme == other.lexeme &&
		this->literal == other.literal && this->line == other.line);


		friend std::ostream& operator<<(std::ostream& os, const Token& token);
		};

Conversation

ujaandas commented Apr 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants