Tool protocol

This page is for the authors of tools that call AL Runner, such as ALchemist and LethAL. It describes the command line, the --output-json document, the coverage files, the serve protocol and the exit codes. For each part it says what differs from AL Runner version 2.

How to use this page: find your tool’s section, apply each “Change”, then run the check at the end of the section.

Why move to version 3

Version 3 reproduces the observable behavior of the AL language, which Microsoft keeps stable for partners. Version 2 depends on Business Central’s internals, which can change with any hotfix or release, so a tool built on version 2 can break without a change on its own side.

1. The command

2. --version

v2 prints ... v2.12.0 ... somewhere in its output. v3 prints

al-runner 3.0.0-alpha.2 (<commit>)

Change: do not match /\bv2\.\d/. Parse the first \d+\.\d+\.\d+ (as ALchemist already does). Read only the first line: when other copies of al-runner are on the PATH, v3 adds one also on PATH: <path>: <its version line> line for each. Both versions start with al-runner (v2 prints al-runner v2.12.0), so a tool that must tell them apart checks for al-runner 3..

3. Flags of al-runner test

Accepted with the v2 meaning unless the “Difference” column says otherwise.

Selection runs whole codeunits: tests of one codeunit share state, so al-runner runs the codeunit of every selected test and reports only the selected tests. A tool that selects one test and excludes its siblings sees the same rows as in v2; it does not save the time of the siblings. With --run-selected-only (in serve: "runSelectedOnly": true) only the selected tests run, as in v2 and as BC’s test tool runs only the test methods that are enabled: the codeunit’s OnRun runs, then the selected tests in declaration order. Globals, rows, time limits and refusals of the other tests then do not reach them.

4. The --output-json document

v2’s shape, pretty printed (the document starts with { in column 0, so the rule “parse from the last { in column 0 to the end” still works). Null and empty fields are left out.

{
  "tests": [
    { "name": "Codeunit50100.Passes", "codeunit": 50100, "codeunitName": "Mixed Tests", "method": "Passes",
      "status": "pass", "durationMs": 12 },
    { "name": "Codeunit50100.Fails", "status": "fail", "message": "boom 5",
      "stackTrace": "\"Mixed Tests\"(CodeUnit 50100).Fails line 3", "durationMs": 4 },
    { "name": "Codeunit50100.Refused", "status": "error", "notRun": true,
      "message": "NotRun: out-of-scope: <reason>" }
  ],
  "passed": 1, "failed": 1, "errors": 1, "skipped": 0, "notRun": 1, "total": 3,
  "exitCode": 1, "wallSeconds": 1.234
}

When the run cannot start (a bad flag, an unreadable app, a missing dependency that stops the load), no document is written, the message is on stderr and the exit code is 2.

4a. Coverage

--coverage-out FILE writes Cobertura XML:

• <coverage ... lines-valid lines-covered>, <sources><source> the working directory, one <package> per app, one <class name=<file name> filename=<path>> per source file, and <lines><line number="N" hits="M" branch="false" />.
• filename is relative to the working directory when the file lies below it, else absolute; always forward slashes. A file inside a package (.app, *.Source.zip) is <package path>!/<path inside>. Line numbers are lines of the file as it is on disk (v2 numbered the instrumented text; v3 does not instrument).
• A source line has one entry when a statement starts on it. hits is the sum of the runs of the statements that start on that line, over the selected tests. A line with no run has hits="0" (v2 left statements nothing reached out in some cases; a tool that scores “no coverage” from a missing line must also treat hits="0" that way).
• There is no <methods> content, no branch data (branch-rate="0").
• Dependency apps’ code is not in the file, only the APPs given on the command line.

--coverage-json FILE writes { "coverage": [...], "perTestCoverage": [...] } in the shapes of the v2 server summary:

{ "coverage": [ { "file": "/abs/path/src/T.al",
                  "statements": [ { "id": 17, "scope": "TakesTheThenBranch", "line": 13, "column": 13, "endLine": 13, "hits": 1 } ] } ],
  "perTestCoverage": [ { "test": "Codeunit50500.TakesTheThenBranch",
                         "coverage": [ { "file": "/abs/path/src/T.al", "statements": [ { "id": 17, "hits": 1, "..." : "..." } ] } ] } ] }

• coverage has every statement of the APPs with its runs (0 when nothing ran it). With --codeunit or --test, the test codeunits left out are not compiled, so their statements are missing; the other objects of the APPs are listed. perTestCoverage has, for each test that ran, only the statements it ran, with its own hits. A test refused while it ran has an entry with the statements it ran before; a test refused before it started has none.
• id is stable inside one run only. scope is the procedure or trigger name. column is a byte column, from 1.
• v2 also sent endColumn; v3 does not know it.
• Selected tests only: a codeunit runs whole, but the tests that --test and --exclude-test leave out do not count.
• What the test ran includes the code its helpers and events ran, in the APPs.

Cost: with coverage off the VM pays one Option check per statement. With it on, one counter bump per statement run.

4b. al-runner serve

Same wire as v2 (protocolVersion 2): one JSON object per line on stdin, answers on stdout, logs on stderr; first stdout line {"ready":true}; command names are case insensitive; [server] request <n> done on stderr after each request except cancel and a line that is not a JSON object. Each request also prints the human Tests: and Result: lines on stderr. Start options are the request-independent options of al-runner test (--package-cache, --dep, --define, --preprocessor-symbols, --test-timeout, --bc-version, --cache-dir, --no-download, --test-data, --no-test-data, --allow-not-run, --server-url, --environment and the like); --protocol 2 is accepted. An APP and the options that select tests or write reports (--filter, --test, --codeunit, --test-exact, --exclude-test, --isolation, --junit, --output-json, --coverage*, --explain) are an error at start-up (exit 2). So are --recipe microsoft and --impact-of: a request runs the tests once, the plain way, and these make a run of another kind.

Differences a client must adapt to:

1. Test lines stream, one per test as it finishes, then the summary.
2. The server is warm and in-process: compiled dependency apps and test data stay in memory. Same sources and the same selected test codeunits as the last request: the compiled workspace is reused and cached is true (a request that selects another codeunit compiles again and has cached: false). A changed source always shows. When only the code of routines of codeunits, tables or table extensions changed, just those objects are compiled again and the summary has incremental: true (a request takes 0.4 s on the corpus app and 2.2 s on Tests-ERM, of which 1.8 s is the tests); any other change compiles the workspace again. Resource files (resourceFolders) are not compiled: each request reads them as they are then, so a changed resource file is seen by the next request and that request still reuses the compile (cached: true). cached says only that the compile was reused. Every request starts from the same post-install database, so nothing leaks between requests. The first request after a new build of al-runner builds the dependency cache (a minute with Microsoft’s apps, then 3 GB resident); with the cache on disk it takes as long as a one-shot run. Repeats take 0.2 to 2 s. A panic in a test is reported as an error for that test and the server goes on.
3. test lines have type, name, codeunit, codeunitName, method (use these instead of splitting name: a method name may hold a dot), status (pass, fail, error), clock on a timed-out test, durationMs, sleptMs (as in --output-json, section 4), message, stackTrace, stackFrames (the AL stack split into lines), and notRun: true for a test that did not run (status error, message NotRun: out-of-scope: <reason>; see section 4). errorKind, alSourceFile, alSourceLine are not sent.
4. The summary has exitCode (the exit code of al-runner test, section 5), passed, failed, errors (= not run), notRun, total, cached, incremental when only some objects were compiled again, protocolVersion, wallSeconds, warnings when there are some, cancelled when cancelled, and coverage / perTestCoverage when asked for (section 4a shapes). selection with affectedOnly, and compilationErrors when the apps do not compile (exit 3). changedFiles, companyInitFailures, sourceScanFailures are not sent.
5. A request that cannot run answers one {"error": "<reason>"} line instead of a summary with exit code 2 or 3: for example {"error":"/x: no such file or folder"}, {"error":"unknown field `cobertura` for runTests"}, {"error":"`sourcePaths` needs at least one app folder"}. A test or excludeTests that matches no test is not an error: it answers a summary with exit code 6, all counts 0, and the reason in warnings. With entries, an entry that matches no test gives the same signal on its entrySummary, and the request’s exit code is 6.
6. While a run is in progress cancel is answered at once. shutdown takes effect after the current test (see the table). Other requests are queued and run in order, as in v2.
7. testIsolation: "test" means TestIsolation Function (section 3), "disabled" runs nothing. "required" (not in v2) runs each test codeunit under its own RequiredTestIsolation, as --isolation required does.

5. Exit codes

Rank when several apply: 2, 6, 1, 8, 7. A tool that reads rows only on exit 0 and 1 must also read rows on 7 (and 8); otherwise every run with a refused test hides all its rows.

6. Environment variables

7. Output on stderr and stdout

v2 printed machine-readable and human lines mixed ([bc] selected BC 27.5, al-runner ... BC ... N apps, [dep] ..., [provision] ..., [pkg-cache] ...). v3 prints none of them. By default it prints the version line (al-runner <version> (<commit>)), the failures, the top refusals, the Tests: line and the Result: line. The setup lines (target: BC <v>, runtime <r>, Microsoft apps from build <b>, build chosen by: ..., dependencies: N apps) are printed only with --verbose. With --output-json and no file, all of it goes to stderr.

Change for a tool that read the [bc] or [dep] lines: treat the BC build and the platform-apps pin as “unknown”, or add --verbose and parse target: BC <v>, ... build <b> from stderr.

8. LethAL (one-shot mode, serverMode: false)

The command LethAL builds (al-runner-transport.ts:641-741) maps to v3 with one rename:

al-runner test --output-json --isolation test --test Codeunit50011.Foo [--exclude-test Codeunit50011.Foo2]...
  [--auto-provision] <sourceDir> [<testDir>] [--package-cache <dir>]... [--define <SYM>]...

Changes in LethAL:

1. Binary: command name al-runner, first argument test. Config key alRunner.alRunnerPath stays.
2. Version check (al-runner-backend.ts:165, 1092-1115; al-runner-contract.ts:439-452): replace /\bv2\.\d/ with a check that stdout starts with al-runner and carries \d+\.\d+\.\d+.
3. Exit codes (al-runner-transport.ts:576-625, 821-845): read the rows on exit 7 and 8 as on 1. Exit 3 means the APPs do not compile, as in v2.
4. Not-run rows (al-runner-backend.ts:97): keep recognizing <Type>: out-of-scope: ...; the v3 message is NotRun: out-of-scope: <reason>. Also read notRun: true. A row with notRun is “runner refused”, never “killed”.
5. Timeouts (al-runner-backend.ts:79-93): unchanged sentence. The limit is CPU time in v3.
6. Compile failure (al-runner-contract.ts fact compile-failure-not-scorable): a mutant that the AL compiler rejects gives exit 3 when the must-not-compile corpus pins the error, else rows with notRun: true. Score both as “not scorable”.
7. Assertion screen (assertion-screen.ts:36-44, 78-90): do not depend on NavNCLDialogException: or InvalidOperationException: out-of-scope:. Messages are the BC text.
8. Stderr (al-runner-transport.ts:106-210): see section 7.
9. Coverage: "coverage": "al-runner" works with --coverage --coverage-out FILE (section 4a). LethAL reads <class ... filename=...> and <line number=... hits=...> by regex: both are as it expects. Labels are relative to the working directory of the process. Per-test coverage (--coverage-json) is new; use it instead of one run per test.
10. Selector: "selectorMode": "static".
11. --auto-provision may stay in the command; it is accepted.

Check: run LethAL’s contract probe against al-runner; the six facts (version, unknown flag rejected with a non-zero exit chosen by the process (2), qualified test name, timeout classified, timeout exit readable, compile failure not scorable) must hold with the changes above.

9. ALchemist

ALchemist needs al-runner serve, execute, value capture and al-runner debug. All four exist; scratch runs, inline values and debugging need the changes below.

1. Binary lookup (alRunnerManager.ts:47-51, 129-139): also which al-runner; spawn al-runner serve instead of al-runner --server (serverProcess.ts:125-141). The version check only warns and accepts 3.0.0 (A3).
2. runtests (lowercase) is accepted. ALchemist sends captureValues and iterationTracking on runTests: v3 puts capturedValues and loops on the test lines (v2 ignored them there). It never sends cobertura or testFilter; both would be refused as unknown fields. Section 4b lists the other differences (warm state, streaming, queued requests).
3. coverage arrives in the summary in the v2 shape (section 4a). ALchemist reads hitStatements and totalStatements, which neither v2 nor v3 sends (a change on the tool’s side).
4. Test names are Codeunit<ID>.<Method> as in v2; the label match in testController.ts:493 (A8) must be checked on a run.
5. Debugger: start al-runner debug --stdio APP... instead of al-runner --dap stdio DIR. Requests, capabilities, events and breakpoint answers are v2’s. Differences: pause stops at the next statement (v2 continued); setExceptionBreakpoints succeeds with no filters; frame names are Object.Routine; a value that is not rendered yet shows <not rendered: KIND>; --test and --codeunit select tests (v2 ran all); a file of a dependency read from a source package is shown from <cache>/debug-sources/ (see Debugging).

A note for ALchemist’s author

The debugger needs one change in ALchemist: the arguments of the DebugAdapterExecutable (debugAdapterFactory.ts:56-98). Pass ['debug', '--stdio', ...sourcePaths] instead of ['--dap', 'stdio', ...sourcePaths], with --dep for a dependency folder whose code should be debugged and the launch configuration’s testFilter as --test. Nothing else in the DAP code has to change: requests, capabilities, events and breakpoint answers are v2’s. ALchemist’s text that says “stepping acts as continue” (debugLaunch.ts:61-62) is out of date: next, stepIn and stepOut step. Debugging, section “Using the debugger from ALchemist”, has the command line and a table of what differs from v2. AL Runner ships no VS Code extension of its own.

10. What does not change for a tool

The names of the tests (Codeunit<ID>.<Method>), the meaning of pass and fail, durationMs, the JUnit file (--junit), the position of positional app folders, --package-cache.