Release v0.14.0

Added

• Latest version includes smart error functionality from the tracer branch that was not included in the last few versions. The latest release includes the smart syntax functionality now that it is as performant as the previous release that did not include smart errors.

• Parser can now be invoked synchronously or asynchronously:

  var sqliteParser = require('sqlite-parser');
  var query = 'select pants from laundry;';
  // sync
  var ast = sqliteParser(query);
  console.log(ast);
  
  // async
  sqliteParser(query, function (err, ast) {
    if (err) {
      console.log(err);
      return;
    }
    console.log(ast);
  });

Changed

• Upgrade sqlite-parser to ES2015

  import sqliteParser from 'sqlite-parser';
  const query = 'select name, color from cats;';
  const ast = JSON.stringify(sqliteParser(query), null, 2);
  console.log(ast);

  • Process is not complete, but as of now most of the parser, tests, and demo are now in ES2015.

• Publish the browserified bundle in the sqlite-parser npm package under dist/ folder

  • This includes the un-minified sqlite-parser.js with sourcemaps and the minified sqlite-parser-min.js without sourcemaps (the default file as defined in the package.json).

• Do not publish the intermediate files from the build process to github

  • The lib/ and dist/ folders are no longer in version control as a part of this github repository.
  • The demo/ folder is also removed from the master branch as well and must be built using grunt demo to use it (or grunt live to build the demo and serve it locally with livereload).

Fixed

• Add --cache flag to pegjs compiler and reduce total rule count to increase performance of tracing parser and smart error functionality.
  • Early results show that --cache makes the tracer parser just as fast as the non-tracer branch for a moderate (~150kB) increase in file size.
  • Removing the number of whitespace rules reduced the chance of the process running out of memory while parsing larger queries.
• Massive reduction in bundled parser size
  • To help combat the extra size added from the --cache option of pegjs, I reduced the size of the parser from 416.89 kB to 86.7 kB (~20% of the original size). I did this by switching pegjs option --optimize from speed to size and modifying [my fork of pegjs)(http://github.com/nwronski/pegjs) to allow rule descriptions to be looked up by rule index instead of by rule name as the optimize size mode required.

Release v0.12.3

Fixed

• Added missing binary division operator so that things like this will now correctly parse.

  select CAST(4 / 9 AS DECIMAL(5,2)) as hat
  from pants;

• Fixed BETWEEN expression grammar to remove bad recursion.

  select num
  from nums n
  where num between 100 AND 200;

• Fixed ORDER BY grammar to allow more than two ordering expressions.

  select color, size, shape, name
  from eggs
  order by color asc, size desc, shape asc

Release v0.11.3

Fixed

• Added missing binary division operator so that things like this will now correctly parse.

  select CAST(4 / 9 AS DECIMAL(5,2)) as hat
  from pants;

• Fixed BETWEEN expression grammar to remove bad recursion.

  select num
  from nums n
  where num between 100 AND 200;

• Fixed ORDER BY grammar to allow more than two ordering expressions.

  select color, size, shape, name
  from eggs
  order by color asc, size desc, shape asc

Release v0.12.2

Fixed

• Refactor to solve the different issues that come from trying to use unquoted reserved words as
  part of table names, column names, aliases, etc... This also addresses issues that came from certain SQLite keywords being fully contained within other keywords (e.g.: IN is contained in INT which is contained in INTERSECT).

  select intersects inid, innot notin
  from fromson nots
  where colorwhere IN nots.pon
  INTERSECT
  select suit, tie from pants;

• Whoops! order property of SELECT statements contained an object with a result key that contained the ordering list instead of just containing the ordering list. It should actually look like this instead:

  {
    "order": [
      {
        "type": "expression",
        "variant": "order",
        "expression": {
          "type": "identifier",
          "variant": "column",
          "name": "hats"
        },
        "direction": "asc"
      }
    ]
  }

Release v0.11.2

Fixed

• Refactor to solve the different issues that come from trying to use unquoted reserved words as
  part of table names, column names, aliases, etc... This also addresses issues that came from certain SQLite keywords being fully contained within other keywords (e.g.: IN is contained in INT which is contained in INTERSECT).

  select intersects inid, innot notin
  from fromson nots
  where colorwhere IN nots.pon
  INTERSECT
  select suit, tie from pants;

• Whoops! order property of SELECT statements contained an object with a result key that contained the ordering list instead of just containing the ordering list. It should actually look like this instead:

  {
    "order": [
      {
        "type": "expression",
        "variant": "order",
        "expression": {
          "type": "identifier",
          "variant": "column",
          "name": "hats"
        },
        "direction": "asc"
      }
    ]
  }

Release v0.12.0

If you want the fastest possible version of the parser, with the tradeoff of poor syntax error feedback, use the v0.12.0 release.

Changed

• all keys removed in all places as it has no effect on query results
• function args property now always contains an array. when DISTINCT is used in function arguments, then a distinct: true property is added to the function node.
• any property that was previously attached to a node with a value of null is no longer included in the AST. this should reduce the size of the AST considerably with useless information. for example, the with property of a SELECT statement node.
• all false values that were included by default (e.g.: temporary, autoIncrement, etc...) are only included in the AST when the value is true
• expected AST for each spec is located in its own .json file instead of keeping it inside of the JS file.

Release v0.11.0

If you want intelligent error messages for syntax errors, at the expense of a substantial decrease in performance, then use the v0.11.0 release.

Changed

• all keys removed in all places as it has no effect on query results
• function args property now always contains an array. when DISTINCT is used in function arguments, then a distinct: true property is added to the function node.
• any property that was previously attached to a node with a value of null is no longer included in the AST. this should reduce the size of the AST considerably with useless information. for example, the with property of a SELECT statement node.
• all false values that were included by default (e.g.: temporary, autoIncrement, etc...) are only included in the AST when the value is true
• expected AST for each spec is located in its own .json file instead of keeping it inside of the JS file.

v0.12.0-beta.1

Changed

• all keys removed in all places as it has no effect on query results
• function args property now always contains an array. when DISTINCT is used in function arguments, then a distinct: true property is added to the function node.
• any property that was previously attached to a node with a value of null is no longer included in the AST. this should reduce the size of the AST considerably with useless information. for example, the with property of a SELECT statement node.
• all false values that were included by default (e.g.: temporary, autoIncrement, etc...) are only included in the AST when the value is true
• expected AST for each spec is located in its own .json file instead of keeping it inside of the JS file.

v0.10.2

Added

• rules and AST for missing transaction-related statement types: RELEASE and SAVEPOINT
• rules and AST for missing SQLite-specific statement types: PRAGMA, DETACH, VACUUM, ANALYZE, and REINDEX
• new specs for SQLite-specific statement types
• new specs for missing transaction-related statement types
• new specs for WITH clause with recursive table expressions
• added new methods in parser-util.js to reduce repeated code: keyify(), textMerge(), and listify()

Changed

• removing Tracer class from sqlite-parser until a faster solution is developed

  • Tracer is causing a 14x performance hit to the sqlite-parser specs when it is enabled
  • might consider having two different builds: one smart error build with Tracer and another performance build for speed

• fixed rules for WITH clause prepended to CRUD-type statements to make sure the with property is added to the correct nodes

• changed the AST for WITH clause to no longer have a node of type "with"

  "with": [
    {
      "type": "expression",
      "format": "table",
      "name": "bees",
      "expression": {
        "type": "statement",
        "variant": "select",
        "from": [],
        "where": null,
        "group": null,
        "result": [],
        "distinct": false,
        "all": false,
        "order": null,
        "limit": null
      },
      "columns": null,
      "recursive": false
    }
  ]

• DROP statement now gives correct variant to the type:'identifier' node in the target property

• now, in a ROLLBACK statement, the savepoint exists on the to property

• fixed bind parameter rules and AST so that a named tcl parameter can still have an alias

• changed the format for INSERT, WITH, and FOREIGN KEY when using a table name versus a table expression name with a column list. for example, INSERT INTO cats (a, b, c) versus INSERT INTO cats now have the following differences in formats

  {
    "into": {
      "type": "identifier",
      "variant": "expression",
      "format": "table",
      "name": "cats",
      "columns": [
        {
          "type": "identifier",
          "variant": "column",
          "name": "a"
        },
        {
          "type": "identifier",
          "variant": "column",
          "name": "b"
        },
        {
          "type": "identifier",
          "variant": "column",
          "name": "c"
        }
      ]
    }
  }

  {
    "into": {
      "type": "identifier",
      "variant": "table",
      "name": "cats",
    }
  }

• JOIN rules so that USING clause can be followed by column names enclosed in parenthesis as the previous rule was not the correct behavior

• JOIN AST modified to have a constraint property, instead of on and using, as a join can only have one of these constraints at a time.

• many places in the AST that previously had a string value in the name property, such as the into property of an INSERT statement, now instead have a node of type 'identifier'

• FOREIGN KEY constraints now have a references property that contains an 'expression' identifier or a 'table' identifier depending on the query instead of the target, columns, and name properties.

• several property values are now being normalized to lowercased strings instead of being passed unmodified to the AST. for example, the action property of action node now contains a lowercased value.

• removed redundant rules that pointed to name rule, such as name_function, name_view, and name_trigger.

• unquoted identifiers are now normalized to lowercased strings as per the SQL-92 standard. quoted identifiers are not normalized.

• SQLite functions are now normalized to lowercase strings in the output AST.

• now preventing FOUC when first loading the demo page. also allowing cursor focus on "Syntax Tree" editor so that the contents can be selected and copied to the clipboard.

• the following things no longer have an identifier node in the name property, as it is too redundant: column constraints, table constrains, column definitions. the parent node provides plenty of context itself for what you will find in its name property.

• lots of clean up to organize tests by category, split out tests to different files and directories by type, and created mocha.opts to run tests directory recursively.

• force update README for npm website

v0.9.8

Added

• new specs for CREATE TRIGGER and datatypes

Changed

• added a bunch of missing descriptions for grammar rules in grammar.pegjs

• make sure that a description is not repeated in smart error message

• comment rules no longer allow you to put a space between the two symbols at the start and/or end of a comment.

  SELECT * - - not valid but is being accepted
  

• added some extra helper rules to CREATE statement rules to help the tracer avoid traversing the wrong create statement type

• allowed characters in identifiers now includes dollar sign $ and no longer includes dash - for unquoted values

• Since SQLite itself is tolerant of this behavior, although it is non-standard, parser allows single-quoted string literals to be interpreted as aliases.

  select
    'hat'.*,
    COUNT(*) as 'pants'
  from
    hats 'hat'

• removed grunt-string-replace from devDependencies

• no longer building demo on top of source in demo/ folder. grunt live now puts assets for interactive demo into .tmp/ folder and then grunt demo creates a min bundle in the demo/ folder

• raw source for interactive demo now exists in src/demo/ folder

• now using grunt-contrib-cssmin to create single css bundle file for demo

Notes

• there is way too much magic/nonsense in the smartError() method of Tracer. need to come up with an alternative approach to getting the right information for syntax errors.