Skip to main content

Interpret automated test output

Last updated on November 5, 2026

Overview​

Automated tests are typically one part of an automated build pipeline, and are used to reduce the cost of a frequent build regime. Automated tests produce two kinds of output:

  • Unreal Log: This log prints in the command line if we run it there, or in the Unreal Log window if we run it in the Unreal Editor. The Unreal log file is simply the output directed to the Unreal console during the session. By default, the log is saved here:

    \<ProjectRoot>\ProjectName\Saved\Logs
  • AccelByte Development Toolkit (ADT) Log files: This test generates an n- number of ADT Log files, each containing test results on one level in JSON lines format. The layout of the ADT Log file is as follows:

    <header>
    ---
    <body>
    ---
    <footer>

When you use Unreal to run automated tests, the system generates an ADT Log file. The path can be anywhere and is used later by the ADT CLI to upload the file. For example:

\<ProjectRoot>\ProjectName\Saved\Logs\AutomatedTest\AutomatedTest_Windows_Level1.json

BlackBox Log files​

Each part of the file is written in JSON lines format: the header is one line, the footer is one line, and each body entry is one line. Put a line with only --- between the header and the body, and between the body and the footer. ADT reads these --- lines to find where each part starts, so keep them in the file.

BlackBox Log file header​

The <header> part of the BlackBox Log File should be:

{"@encoding": "blackbox/snifftest+json.v2.1.0", "test_id": "<test id in UUID>", "map": "<map>"}
  • @encoding is the encoding version of this ADT Log file. Use blackbox/snifftest+json.v2.1.0. The ADT CLI checks this value when you upload the file, and ADT Web uses it to read the file.
  • test_id is a generated random UUID that identifies this test run. Use the same value in every body line and in the footer.
  • map is the map (level) where the current test is running. ADT Web shows it in the Automated Test list.

BlackBox Log file body​

The <body> part of the test result should be in the following format:

{"test_id": "<test id in UUID>", "type": "begin_test", "name": "test1"}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "error", "msg": "errors", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (777)", "timestamp": 1612416088}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "warning", "msg": "warning", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (778)", "timestamp": 1612416089}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "info", "msg": "info", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (779)", "timestamp": 1612416090}
{"test_id": "<test id in UUID>", "type": "end_test", "name": "test1", "result": "success"}

For example:

{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "Mantle"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : start", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (777)", "timestamp": 1612416088}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition Teleport to Mantle1", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (830)", "timestamp": 1612416089}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : Mantle state reached", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (877)", "timestamp": 1612416090}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition from actor (Mantle1) : Mantle Success", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (884)", "timestamp": 1612416091}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : finish", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (892)", "timestamp": 1612416092}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "Mantle", "result": "success"}

Wrap logs inside "begin_test" and "end_test". The test fails if there are error-type logs inside.

The body of "begin_test" type consists of these parameters:

  • test_id of the test run, the same value as in the header
  • type of the current JSON line, in the format "type":"begin_test"
  • name of the current test in the format "name": "test1"

The body of "msg" type consists of these parameters:

  • test_id of the test run, the same value as in the header
  • type of the current JSON line, in the format "type":"msg"
  • msg_type is the log verbosity type (info, error, warning)
  • msg is the log message
  • callstack is the call stack where the log is called from
  • timestamp is the time the log was written, in Unix time (seconds, not milliseconds). Add it to every "msg" line. ADT Web shows it next to the log. If it is missing, the log still shows, but without a time. ADT Web lists logs in the order they appear in the file, not by timestamp.

The body of "end_test" type consists of these parameters:

  • test_id of the test run, the same value as in the header
  • type of the current JSON line, in the format "type":"end_test"
  • name of the current test
  • result of the current test, "failed" if there are error log messages.

The <footer> part of the test result:

{"test_id": "<test id in UUID>", "total_test": 0, "failed": 0, "success": 0, "result": "<result>"}
  • test_id of the test run, the same value as in the header
  • total_test is the number of tests in the <body> part (each "begin_test" and "end_test" pair is one test). Write total_test, failed, and success as JSON numbers (for example, 2), not as strings (for example, "2"). The 0 values in the template above are placeholders.
  • failed is the number of failed tests
  • success is the number of successful tests
  • result is the overall result of the test run. The options are success or failed. The result is success only if there are no failed results for any of the records inside the body.

Example of a test result​

Example of an ADT Log file using a game called ABShooter in Level 1:

{"@encoding": "blackbox/snifftest+json.v2.1.0", "test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "map": "Level_1"}
---
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "LoadMap-Level_1"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "warning", "msg": "RegisterLocationMarkersForBodyPartCaps has been called", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416088}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "warning", "msg": "GetVertexLocations returned false for mesh HeadStump.", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416089}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "FlushAsyncLoading: 1 QueuedPackages, 0 AsyncPackages", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416090}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "ULevelStreaming::RequestLevel(/Game/Maps/Game/Escape/Sections/Security/Escape_Security_VFX) is flushing async loading", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416091}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "error", "msg": "USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416092}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "error", "msg": "USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416093}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "LoadMap-Level_1", "result": "failed"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "Flashlight-PlayerStart"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "Flashlight test : start", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416094}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "Flashlight test : Success", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416095}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "Flashlight-PlayerStart", "result": "success"}
---
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "total_test": 2, "failed": 1, "success": 1, "result": "failed"}