ArkScript v4.0.0

[SuperFola]

Changes for you, as a user

• macros are defined using (macro name value) and (macro name (foo bar ...args) body) instead of !{name value} and !{name (foo bar ...args) body}
• quote keyword got removed. You can emulate it with a macro: (macro quote (value) (fun () value)) (it wasn't really a Lisp quote in the first place...)
• while loops have their own scope
• imports changed a lot: from (import "path/to/file.ark") to java like imports (the root is deduced from the main script emplacement): (import path.to.file :foo :bar)
  • for more details, see the documentation
• you can format your code using the cli: arkscript -f myfile.ark
• new dict datatype: https://arkscript-lang.dev/docs/std/dict/

For more information, check the language documentation!

Technical changelog

Added

• more tests for the io builtins
• added lines and code coloration in the error context
• new dependency: fmtlib
• added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
• $repr macro to get a string representation of a given node
• added boost-ext/ut to write unit tests in C++
• basic ArkScript code formatter, available through the CLI: arkscript -f|--format
• comments are now tracked in the AST and attached to the nearest node below them
• VM::forceReloadPlugins, to be used by the REPL to force reload the plugins and be sure that their symbols are all defined
• added help, save (save history to disk), history (print history), reset (reset vm and code) commands to the REPL
• REPL can now show when a code block isn't terminated (prompt changes from > to :)
• more controls available inside the REPL
• fuzzing step in the CI
• better error reporting on unknown import
• check on the number of arguments passed to type
• warning when the formatter deletes comment(s) by mistake
• check on arguments passed to list, concat, append and friends to only push valid nodes (that produce a value)
• introduced Ark::internal::Pass to describe compiler passes: they all output an AST (parser, import solver, macro processor, and optimizer for now)
• add -f(no-)importsolver, -f(no-)macroprocessor and -f(no-)optimizer to toggle on and off those compiler passes
• added resolving empty? as a macro when possible
• added short-circuiting to and and or implementation
• added --check to the formatter as an option: returns 0 if the code is correctly formatted, 1 otherwise
• the name & scope resolution pass now checks for mutability errors
• compile-time checks for mutability errors with append!, concat! and pop!
• new MAKE_CLOSURE <page addr> instruction, generated in place of a LOAD_CONST when a closure is made
• added -fdump-ir to dump the IR entities to a file named {file}.ark.ir
• added 11 super instructions and their implementation to the VM
• support for the glob import syntax and symbol import syntax
• modify list and return a copy (string:setAt string index char) (bound checked)
• added in place list mutation: (@= list|string index new_value), (@@= list|list<string> index1 index2 new_value|char) (bound checked)
• compile time argument count check for and and or
• basic dead code elimination in the AST optimizer
• new operator @@ to get elements in list of lists / list of strings
• new builtin random, returning a random number between INT_MIN and INT_MAX, or in a custom range
• $as-is to paste a node inside a maro without evaluating it further; useful to stop recursive evaluation of nodes inside function macros
• LOAD_SYMBOL_BY_INDEX instruction, loading a local from the current scope by an index (0 being the last element added to the scope)
• STORE_FROM_INDEX and SET_VAL_FROM_INDEX instructions for parity with the super instructions not using load by index
• INCREMENT_BY_INDEX and DECREMENT_BY_INDEX instructions for parity with the super instructions not using load by index
• STORE_TAIL_BY_INDEX, STORE_HEAD_BY_INDEX, SET_VAL_TAIL_BY_INDEX, SET_VAL_HEAD_BY_INDEX super instructions added for parity with the super instructions not using load by index
• RESET_SCOPE_JUMP instruction emitted at the end of a while loop to reset a scope so that we can create multiple variables and use LOAD_SYMBOL_BY_INDEX
• instruction source location; two new bytecode tables were added: one for filenames, another for (page pointer, instruction pointer, file id, line), allowing the VM to display better error messages when the source is available
• show source location when a runtime error is thrown in the VM
• LT_CONST_JUMP_IF_FALSE and LT_SYM_JUMP_IF_FALSE to compare a symbol to a const and a symbol to a symbol (respectively), then jump to an address if false (useful for while loops that check a simple (< x n) condition)
• LT_CONST_JUMP_IF_TRUE, counterpart of LT_CONST_JUMP_IF_FALSE
• GT_CONST_JUMP_IF_TRUE, counterpart of LT_CONST_JUMP_IF_TRUE
• GT_CONST_JUMP_IF_FALSE, counterpart of LT_CONST_JUMP_IF_FALSE
• GT_SYM_JUMP_IF_FALSE, counterpart of LT_SYM_JUMP_IF_FALSE
• CALL_SYMBOL super instruction to load and call a symbol in a single instruction
• GET_FIELD_FROM_SYMBOL and GET_FIELD_FROM_SYMBOL_INDEX super instructions to get a field from a closure and push it to the stack
• EQ_CONST_JUMP_IF_TRUE and EQ_SYM_INDEX_JUMP_IF_TRUE to compare a symbol to a const and a symbol to a symbol (respectively), then jump to an address if true (useful for conditions that check a simple (= x n) condition)
• NEQ_CONST_JUMP_IF_TRUE as a super instruction counterpart to EQ_CONST_JUMP_IF_TRUE
• NEQ_SYM_JUMP_IF_FALSE, counterpart of LT_SYM_JUMP_IF_FALSE for inequality
• AT_SYM_SYM and AT_SYM_INDEX_SYM_INDEX super instructions, to get an element from a list in a single instruction, avoiding 2 push and 2 pop
• CHECK_TYPE_OF and CHECK_TYPE_OF_BY_INDEX super instructions, to check the type of variable against a constant in a single instruction
• INCREMENT_STORE and DECREMENT_STORE super instructions, to update a value in place when incrementing/decrementing it by a set amount
• APPEND_IN_PLACE_SYM and APPEND_IN_PLACE_SYM_INDEX super instructions
• PUSH_RETURN_ADDRESS instruction now replaces the VM auto push of IP/PP
• remove the stack swapping by pushing arguments in the reverse order by which they are loaded
• wasm export: we can now run ArkScript code on the web!
• GET_CURRENT_PAGE_ADDRESS instruction to push the current page address to the stack
• CALL_CURRENT_PAGE super instruction, calling the current page with a given number of arguments (avoid loading a page address on the stack, then popping it to perform the call)
• new data type Dict, which can be created with (dict "key" "value" ...), and manipulated with dict:get, dict:add, dict:contains, dict:remove, dict:keys and dict:size
• added program name under `builtin__sys:programName
• STORE_LEN super instruction, to load a symbol by index and store its length (if it's a string or list) in a new variable
• AT_SYM_INDEX_CONST super instruction, to load a value from a container using a constant as the index

Changed

• instructions are on 4 bytes: 1 byte for the instruction, 1 byte of padding, 2 bytes for an immediate argument
• enhanced the bytecode reader and its command line interface
• added the padding/instruction/argumentation values when displaying instructions in the bytecode reader
• fixed underline bug in the error context
• the str:format functions now expect strings following this syntax: https://fmt.dev/latest/syntax.html
• more documentation about the compiler implementation
• more documentation about the virtual machine
• closures can be now be compared field per field: (= closure1 closure2) will work only if they have the same fields (name) and if the values match
• macros are now defined like (macro name value) / (macro name (args args args) body) / ($if cond then else)
• upgraded from C++17 to C++20
• new parser, new syntax for imports: (import package.sub.file)
• allow nodes to be empty when dumping the AST to JSON
• Macros can be declared inside a begin block within a cond macro and used in the scope surrounding the cond macro
• arkscript --version and arkscript --help now output ArkScript version with the commit hash
• void Value::toString(std::ostream&, VM&) now becomes std::string Value::toString(VM&)
• removed Node::operator<< to replace it with Node::debugPrint
• fixed a bug in the compiler where one could pass a non-symbol to let, mut or set, resulting in a compiler crash
• fixed a bug in the macro processor where one could pass an unknown symbol to argcount and crash the processor
• fixed a bug in the compiler where one could pass something other than a list to (fun) as the argument block, resulting in a crash
• fixed a bug in the compiler generating non-callable functions
• fixed a bug in the macro processor generating invalid let / mut / set nodes
• fixed a bug in the macro processor allowing out-of-bounds access with (macro test (@ [1 2 3] -5))
• fixed a bug in the VM which wrongfully allowed self concat in place: (concat! lst lst)
• fixed a bug in the compiler where one could "use" operators without calling them: (print nil?)
• fixed a bug in the compiler allowing the use of operators without any argument: (+)
• fixed a bug in the VM during error reporting when a non-function was used as a function
• refactored code inside the bytecode reader to promote code reuse
• fixed a bug in the compiler generating invalid while nodes
• fixed a bug when passing the wrong number of arguments to a function inside an async call was crashing the VM because the function couldn't be named
• fixed a bug in the compiler generating invalid fun nodes
• fixed a bug when generating let, mut or set nodes inside macros with an invalid node type
• fixed a bug when reading invalid UTF8 codepoints in the parser caused out-of-bounds reads
• fixed a bug with recursive macros, exhausting the stack space due to recursive evaluation
• Futures can be awaited again; they will return nil on all the tries
• checking for reused argument name in macros during parsing
• enhanced comment after node handling in macros
• adding a hard limit on package names' length (255 characters, to comply with POSIX limits)
• disallow passing invalid nodes as arguments to functions and operators
• checking for unevaluated spread inside macros
• checking for invalid symbols when defining a function through a macro
• added a max macro unification depth (256)
• added a max macro evaluation depth (256)
• introduced internal::listInstructions with the different instructions, to be used by the compiler and name resolution pass
• checking for forbidden variable/constant name in the name & scope resolution pass, to give errors to the user before compiling some weird code
• repl completion and colors are now generated automatically from the builtins, keywords & operators
• fixed formatting of comments inside function declarations
• renamed the macros symcat and argcount to $symcat and $argcount for uniformity
• the Ark::VM class is now final
• the STORE instruction has been renamed SET_VAL
• the STORE instruction is emitted in place of the LET and MUT instructions, without any mutability checking now
• io:writeFile no longer takes a mode and has been split into io:writeFile and io:appendToFile
• instructions are now positioned like this: inst byte1 byte2 byte3
  • byte1 is 0 if the instruction takes a single argument on 16 bits, split on byte2 and byte3
  • if the instruction takes two arguments, they each have 12 bits ; the second one is on byte1 and the upper half of byte2, the first on lower half of byte2 and then byte3
• ast-to-json dump now supports macros
• The parser can detect ill-formed macros (that are seen as function macros while being value macros)
• adding a CALL_BUILTIN <builtin> <arg count> super instruction
• fixed formatting of comments after the last symbol in an import node
• renamed str:xyz builtins to string:xyz for uniformity with the standard library
• string:find takes an optional third argument, startIndex (where to start the lookup from, default 0
• list:setAt can work with negative indexes, and is now bound checked
• re-enabled the AST optimizer, only used for the main arkscript executable (not enabled when embedding arkscript, so that one can grab variables from the VM)
• loops have their own scope: variables created inside a loop won't leak outside it
• upgraded fmtlib to 11.1.3-13
• allow capture in nested scope (before it was targeting only the current scope)
• -bcr option can be given a source file, it will then be compiled before its bytecode is shown
• magic numbers for tables start in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xA1, 0xA2, 0xA3 (symbols, values, code) to make them stand out in hex editors
• magic numbers for value types in bytecode files have been changed from 0x01, 0x02, 0x03 to 0xF1, 0xF2, 0xF3 (number, string, function)
• numbers in the values table in bytecode files are no longer turned to string, but their IEEE754 representation is now encoded on 12 bytes (4 for the exponent, 8 for the mantissa)
• changed how scopes are stored inside the VM to enhance performance. All scope data are now contiguous!
• when possible, accessing variables from the current scope is compiled to a new instruction LOAD_SYMBOL_BY_INDEX, to avoid the sometimes expensive lookup by id
  • This works inside normal scopes (introduced by while loops) and function scopes, but not for closures
• VM stack size is now 4096 instead of 8192
• Ark::CodeError now takes a CodeErrorContext to store the source (filename, line, column, expression) of an error
• renamed string:format to format
• io:removeFiles is now io:removeFile and works on a single file/path
• renamed almost all builtins to prefix them with builtin__, to have them proxied in the standard library (to be able to import and scope them properly)
• new super instruction CALL_BUILTIN_WITHOUT_RETURN_ADDRESS to optimize the proxied builtins, skipping the return address deletion
• The VM no longer stores a reference to the current function being called in the newly created scope
• execution contexts can be reused for async calls if they are not active, to avoid constantly requesting memory and creating (heavy) contexts
  • If there are more than 5 contexts, the 6th one will be destroyed once it completes
• execution contexts are now marked as free to be reused (or deleted) once a value has been computed, without waiting for a call to await
• captures are not renamed anymore by the NameResolutionPass (which used to fully qualify captured names when possible, which isn't desirable: when you capture &foo, you expect to be able to use .foo not .module:foo)
• when loading a module, its mappings are loaded in the current scope instead of the global scope
• argument order in the CLI changed: the file to run (and its optional script arguments) are now last, to be more consistent with all the other existing tooling (Python, Docker...)
• VM stack size has been upped to 4096 + 256, to have a buffer to be able to catch stack overflows without hindering performances too much
• can not create a variable in a function shadowing said function, to prevent weird bugs when trying to do recursion for example

Removed

• removed unused NodeType::Closure
• removing the custom string, replacing it with std::string (the format engine of the custom string had a lot of memory leaks)
• Utils::digPlaces and Utils::decPlaces got removed as they were no longer needed
• removed deprecated code (list:removeAt, ark executable now replaced by arkscript)
• removed VM::getUserPointer and VM::setUserPointer
• removed ARK_PROFILER_COUNT define
• removed useless \0 escape in strings
• removed termcolor dependency to rely on fmt for coloring outputs
• removed and and or instructions in favor of a better implementation to support short-circuiting
• removed LET and MUT instructions in favor of a single new STORE instruction
• removed SAVE_ENV instruction
• removed Value VM::resolve(const Value* val, Args&&... args), which has been deprecated in ArkScript v3.4.0

---

This discussion was created from the release ArkScript v4.0.0.