Whitespace and Line Terminators

Status: implemented; JavaScript's other whitespace is refused, see the page

Whitespace is a space (U+0020) or a tab (U+0009). A newline is a line feed (LF, U+000A), a carriage return (CR, U+000D), or CR followed by LF, which is one line break. Those are JSON's four whitespace characters, the same four DataJS admits.

Trivia may stand between any two tokens, before the first and after the last, and it is insignificant there, newlines included, but for one: a statement written without its ; ends at the newline before the next statement (module structure). Inside an expression a line break reads as a space does, exactly as JavaScript reads it: a followed by .length, [0] or + 2 on the next line is a.length, a[0] or a + 2, and f followed by (1) is the call f(1). Trivia is needed only where two tokens would otherwise read as one: const a, not consta (module structure); 1 .x, not 1.x (property access).

The exceptions are the boundaries JavaScript restricts, before => and after return and throw, where a line break is refused (functions). There the trivia is a line break if a newline of any of the three forms stands in it, inside a block comment or not (comments).

Nothing else is trivia. JavaScript's other whitespace — VT (U+000B), FF (U+000C), U+FEFF, and the Unicode space separators such as NBSP (U+00A0) and U+3000 — is refused, although JavaScript accepts it, a byte order mark at the start of the file included (source text). JavaScript's other two line terminators, U+2028 and U+2029, may stand raw only inside a string, as in JSON; anywhere else, a comment included, they are refused, since an invisible line break is no line break here. A hashbang comment, #! at the start of the file, is refused too, although JavaScript admits one (hash comments). Any other character outside a string or a comment is part of a token or an error, as it is in JavaScript.

Examples

The compiler demo offers the programs whitespace in module.f.js, each with its real output: one accepted, one refused with the compiler's own message.

The demo on this page runs sample.f.js, this page's example as a function of one number — tabs, spaces and line breaks between the tokens of one expression — with every output fjs compile makes of it: the expression graph (EDAG), the Rust and the WebAssembly that Rust builds to. Its Run button calls the default export with 1 and 2, as JavaScript and as WebAssembly, and shows what each call returned or threw.

export default (...a) =>
	a[0]
    +
	1;

Reference

Demo

In FunctionalScript

fjs compile accepts this module, and it runs as JavaScript and as WebAssembly alike.

A FunctionalScript module and what fjs compile makes of it: the expression graph it lowers to, the Rust it compiles to on the NaNVM runtime, and the WebAssembly that Rust builds to. Run calls its default export with 1, then 2, first as the JavaScript it already is and then as WebAssembly, and shows what each call returned or threw and how long one call takes, timed over 1000 calls.

FunctionalScript source:

/**
 * The example of [`README.md`](./README.md) (Whitespace and Line
 * Terminators) as a function of one number, for the page's WASM demo: tabs,
 * spaces and line breaks between the tokens of one expression.
 *
 * @module
 */

/** @type {(...a: readonly number[]) => unknown} */
export default (...a) =>
	a[0]
    +
	1;

Expression graph (EDAG) that fjs compile lowers it to:

=>lengthbody0++leftright.1.objidxrest0

The same graph, as fjs compile writes it to a .edag.data.js file:

export default ["{}",[[":","default",["=>",0,[],["+",[".",["rest"],0],1]]]]];

Rust that fjs compile writes from it, on the NaNVM runtime:

// @generated by `fjs compile`. Do not edit: recompile the source module instead.

use nanvm_lib::vm::unstable::{f64_any, string_key};
use nanvm_lib::vm::{Any, Array, IStaticFunction, ToAny, ToObject};

#[rustfmt::skip]
#[allow(clippy::if_same_then_else)]
pub fn module<A: IStaticFunction>(_runtime: Any<A>) -> Result<Any<A>, Any<A>> {
    Ok([(string_key("default"), A::static_function(|_self, _args| {
        let _b132: Any<A> = _args.rest(0).to_any();
        (Any::dot(_b132.clone(), f64_any(0x0000000000000000)).end())? + f64_any(0x3ff0000000000000)
    }, 0, Array::default(), Some("(...$0)=>$0[0]+1")).to_any())].to_object().to_any())
}

WebAssembly text (WAT) of the functions compiled from this source, without the NaNVM runtime:

  (func $nanvm_demos::samples::whitespace::module::<nanvm_lib::naive::Naive> (;394;) (type 1) (param i32 i32)
    (local i32 i32 i32 i32 i32)
    global.get $__stack_pointer
    i32.const 96
    i32.sub
    local.tee 2
    global.set $__stack_pointer
    local.get 2
    i32.const 0
    i32.store16 offset=28
    local.get 2
    i32.const 1062452
    i32.store offset=24
    local.get 2
    i32.const 1062445
    i32.store offset=20
    local.get 2
    i32.const 48
    i32.add
    local.get 2
    i32.const 20
    i32.add
    call $"core::iter::adapters::try_process::<core::iter::adapters::map::Map<core::str::iter::EncodeUtf16, core::result::Result<u16, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>::Ok>, u16, core::result::Result<core::convert::Infallible, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>, <core::result::Result<alloc::rc::Rc<[u16]>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>> as core::iter::traits::collect::FromIterator<core::result::Result<u16, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>>>::from_iter<core::iter::adapters::map::Map<core::str::iter::EncodeUtf16, core::result::Result<u16, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>::Ok>>::{closure#0}, alloc::rc::Rc<[u16]>>"
    block ;; label = @1
      block ;; label = @2
        block ;; label = @3
          local.get 2
          i32.load8_u offset=48
          local.tee 3
          i32.const 255
          i32.ne
          br_if 0 (;@3;)
          local.get 2
          i32.load offset=56
          local.set 4
          local.get 2
          i32.load offset=52
          local.set 5
          local.get 2
          i32.const 0
          i32.store8 offset=32
          local.get 2
          i32.const 8
          i32.add
          local.get 2
          i32.const 32
          i32.add
          call $"<core::iter::adapters::GenericShunt<core::iter::adapters::map::Map<core::iter::sources::empty::Empty<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>, core::result::Result<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>, ()>::Ok>, core::result::Result<core::convert::Infallible, ()>> as alloc::rc::ToRcSlice<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>>>::to_rc_slice"
          local.get 2
          local.get 2
          i32.load offset=12
          local.tee 6
          i32.store offset=52
          local.get 2
          local.get 2
          i32.load offset=8
          local.tee 3
          i32.store offset=48
          block ;; label = @4
            local.get 2
            i32.load8_u offset=32
            i32.const 1
            i32.ne
            br_if 0 (;@4;)
            local.get 3
            local.get 3
            i32.load
            i32.const -1
            i32.add
            local.tee 0
            i32.store
            local.get 0
            br_if 3 (;@1;)
            local.get 2
            i32.const 48
            i32.add
            call $"<alloc::rc::Rc<[nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>]>>::drop_slow"
            br 3 (;@1;)
          end
          local.get 3
          i32.eqz
          br_if 2 (;@1;)
          i32.const 82
          i32.const 0
          local.get 3
          local.get 6
          i32.const 1062452
          i32.const 16
          call $<nanvm_lib::naive::function::Function>::new
          local.set 3
          local.get 2
          i32.const 0
          i32.store8 offset=20
          local.get 2
          local.get 3
          i32.store offset=68
          local.get 2
          i32.const 7
          i32.store8 offset=64
          local.get 2
          local.get 4
          i32.store offset=60
          local.get 2
          local.get 5
          i32.store offset=56
          local.get 2
          i64.const 4294967296
          i64.store offset=48
          local.get 2
          local.get 2
          i32.const 20
          i32.add
          i32.store offset=80
          local.get 2
          local.get 2
          i32.const 48
          i32.add
          call $"<core::iter::adapters::GenericShunt<core::iter::adapters::map::Map<core::array::iter::IntoIter<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>), 1>, core::result::Result<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>), ()>::Ok>, core::result::Result<core::convert::Infallible, ()>> as alloc::rc::ToRcSlice<(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>)>>::to_rc_slice"
          local.get 2
          local.get 2
          i32.load offset=4
          local.tee 4
          i32.store offset=36
          local.get 2
          local.get 2
          i32.load
          local.tee 3
          i32.store offset=32
          block ;; label = @4
            local.get 2
            i32.load8_u offset=20
            i32.const 1
            i32.ne
            br_if 0 (;@4;)
            local.get 3
            local.get 3
            i32.load
            i32.const -1
            i32.add
            local.tee 0
            i32.store
            local.get 0
            br_if 2 (;@2;)
            local.get 2
            i32.const 32
            i32.add
            call $"<alloc::rc::Rc<[(nanvm_lib::vm::string::String<nanvm_lib::naive::Naive>, nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>)]>>::drop_slow"
            br 2 (;@2;)
          end
          local.get 3
          i32.eqz
          br_if 1 (;@2;)
          local.get 0
          local.get 4
          i32.store offset=16
          local.get 0
          local.get 3
          i32.store offset=12
          local.get 0
          i32.const 5
          i32.store8 offset=8
          local.get 0
          i64.const 0
          i64.store
          local.get 1
          call $"#func386 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 2
          i32.const 96
          i32.add
          global.set $__stack_pointer
          return
        end
        local.get 2
        local.get 2
        i32.load8_u offset=51
        i32.store8 offset=35
        local.get 2
        local.get 2
        i32.load16_u offset=49 align=1
        i32.store16 offset=33 align=1
        local.get 2
        local.get 2
        i32.load offset=60
        i32.store offset=44
        local.get 2
        local.get 2
        i64.load offset=52 align=4
        i64.store offset=36 align=4
        local.get 2
        local.get 3
        i32.store8 offset=32
        i32.const 1065572
        i32.const 43
        local.get 2
        i32.const 32
        i32.add
        i32.const 1065556
        i32.const 1065616
        call $core::result::unwrap_failed
        unreachable
      end
      i32.const 1057428
      i32.const 43
      local.get 2
      i32.const 95
      i32.add
      i32.const 1057412
      i32.const 1057472
      call $core::result::unwrap_failed
      unreachable
    end
    i32.const 1057428
    i32.const 43
    local.get 2
    i32.const 95
    i32.add
    i32.const 1057412
    i32.const 1057472
    call $core::result::unwrap_failed
    unreachable
  )
  (func $"<nanvm_demos::samples::whitespace::module<nanvm_lib::naive::Naive>::{closure#0} as core::ops::function::FnOnce<(&nanvm_lib::naive::function::Function, nanvm_lib::vm::array::Array<nanvm_lib::naive::Naive>)>>::call_once" (;395;) (type 4) (param i32 i32 i32 i32)
    (local i32 i32)
    global.get $__stack_pointer
    i32.const 128
    i32.sub
    local.tee 4
    global.set $__stack_pointer
    local.get 4
    local.get 3
    i32.store offset=12
    local.get 4
    local.get 2
    i32.store offset=8
    block ;; label = @1
      local.get 2
      i32.load
      local.tee 5
      i32.const -1
      i32.eq
      br_if 0 (;@1;)
      local.get 4
      local.get 3
      i32.store offset=24
      local.get 4
      local.get 2
      i32.store offset=20
      local.get 4
      i32.const 6
      i32.store8 offset=16
      local.get 2
      local.get 5
      i32.const 2
      i32.add
      local.tee 5
      i32.store
      local.get 5
      i32.eqz
      br_if 0 (;@1;)
      local.get 4
      local.get 2
      i32.store offset=116
      local.get 4
      i32.const 6
      i32.store8 offset=112
      local.get 4
      local.get 3
      i64.extend_i32_u
      i64.store offset=120
      local.get 4
      i64.const 0
      i64.store offset=88
      local.get 4
      i32.const 2
      i32.store8 offset=80
      local.get 4
      i32.const 48
      i32.add
      local.get 4
      i32.const 112
      i32.add
      local.get 4
      i32.const 80
      i32.add
      call $<nanvm_lib::vm::lambda::member::Member<nanvm_lib::naive::Naive>>::new
      block ;; label = @2
        block ;; label = @3
          block ;; label = @4
            local.get 4
            i32.load8_u offset=48
            i32.const 255
            i32.ne
            br_if 0 (;@4;)
            local.get 0
            local.get 4
            i64.load offset=64
            i64.store offset=16
            local.get 0
            local.get 4
            i64.load offset=56
            i64.store offset=8
            local.get 0
            i64.const 1
            i64.store
            local.get 4
            i32.const 16
            i32.add
            call $"#func386 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
            local.get 2
            local.get 2
            i32.load
            i32.const -1
            i32.add
            local.tee 3
            i32.store
            local.get 3
            i32.eqz
            br_if 1 (;@3;)
            br 2 (;@2;)
          end
          local.get 4
          local.get 4
          i64.load offset=72
          i64.store offset=104
          local.get 4
          local.get 4
          i64.load offset=64
          i64.store offset=96
          local.get 4
          local.get 4
          i64.load offset=56
          i64.store offset=88
          local.get 4
          local.get 4
          i64.load offset=48
          i64.store offset=80
          local.get 4
          i32.const 112
          i32.add
          local.get 4
          i32.const 80
          i32.add
          call $<nanvm_lib::vm::lambda::member::Member<nanvm_lib::naive::Naive>>::own
          block ;; label = @4
            block ;; label = @5
              local.get 4
              i32.load8_u offset=112
              i32.const 255
              i32.eq
              br_if 0 (;@5;)
              local.get 4
              local.get 4
              i64.load offset=120
              i64.store offset=40
              local.get 4
              local.get 4
              i64.load offset=112
              i64.store offset=32
              br 1 (;@4;)
            end
            local.get 4
            i32.const 256
            i32.store16 offset=32
          end
          local.get 4
          i32.const 80
          i32.add
          call $"#func562 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 4
          i32.const 96
          i32.add
          call $"#func562 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 4
          i64.const 4607182418800017408
          i64.store offset=88
          local.get 4
          i32.const 2
          i32.store8 offset=80
          local.get 0
          local.get 4
          i32.const 32
          i32.add
          local.get 4
          i32.const 80
          i32.add
          call $"<nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive> as core::ops::arith::Add>::add"
          local.get 4
          i32.const 16
          i32.add
          call $"#func386 core::ptr::drop_glue::<nanvm_lib::vm::unpacked::Unpacked<nanvm_lib::naive::Naive>>"
          local.get 2
          local.get 2
          i32.load
          i32.const -1
          i32.add
          local.tee 3
          i32.store
          local.get 3
          br_if 1 (;@2;)
        end
        local.get 4
        i32.const 8
        i32.add
        call $"<alloc::rc::Rc<[nanvm_lib::vm::any::Any<nanvm_lib::naive::Naive>]>>::drop_slow"
      end
      local.get 4
      i32.const 128
      i32.add
      global.set $__stack_pointer
      return
    end
    unreachable
  )

Emergent Testing

Idle. Press Run to start the suite.

  • ./proof.f.mjs