syntax-sandbox

This repository contains my experiment on creating a query language for the CodeMirror. It is a demo, not a reusable package.

The application demonstrates how to create custom Query Language for CodeMirror using the Lezer Parser System. It supports autocomplete, syntax highlight, validation, and custom UI decorations.

My goal was to create query language that is unbound from an application domain. So the grammar part defines just a syntax for queries, but does not contains any properties or values. Thus, domain specific data can be defined independently of grammar.

Architecture

At the application level users of the component describe available properties and values. Please see the example in src/data. They can also define a Suggest and/or Validator classes to implement some custom logic. Then they need to instantiate a data model with defined properties and pass it to the Editor, see src/main.ts. Optionally, they can also create a Decorator to add some specific UI features for different properties and values: src/decorator.

Then, the Editor renders a CodeMirror instance with some plugins. In the src/language multiple CodeMirror plugins are instantiated using defined data model and decorator. The parser is generated using Lezer from the QueryLanguage.grammar. The mapper from syntax tree to the MongoDB-like JSON is implemented in src/language/query.

The Editor emits change event with the result of parsing. The result is shaped as MongoDB-like JSON to show the query tree.

What can be improved:

• Asyncronous suggest for data-intensive applications
• More operators, for example for numeric fields
• Better UX for different kind of values, for example datepicker for date and time

Commands

Start:

npm run dev

Rebuild grammar:

npm run grammar

Run tests:

npm test

File Structure

1. main.ts is an entrypoint. It finds the root DOM element and render the app.
2. Domain Specific Data
   1. data contains some data and declarations that are related to a specific app (e.g. task manager)
      1. properties describes possible properties and its values available for QL
      2. persons contains some imaginable “users” that could be used for filtering by an author
   2. decorator defines how to replace query tokens with rich HTML widgets, e.g. it replaces usernames of authors with their names
3. Universal query language: language contains a set of CodeMirror plugins that implement work with the query language defined in Grammar
   1. grammar contains everything about parsing a text to an AST
      1. QueryLanguage.grammar grammar definition in terms of The Lezer Parser System. The parser itself is autogenerated based on the grammar file.
   2. query a helper to transform AST to query tree, it helps to visualize the parsing result and test the parser
   3. highlighter highlights different semantic tokens with different colors
   4. linter shows errors for improper tokens
   5. decorator shows user-defined widgets for some tokens
   6. autocomplete suggests related tokens to a user
   7. language combines them all