Interpret automated test output

Last updated on August 28, 2023

Overview​

Automated tests are typically one part of an automated build pipeline, and are used to reduce the cost of a frequent build regime. Automated tests produce two kinds of output:

  • Unreal Log: This log prints in the command line if we run it there, or in the Unreal Log window if we run it in the Unreal Editor. The Unreal log file is simply the output directed to the Unreal console during the session. By default, the log is saved here: 

    \<ProjectRoot>\ProjectName\Saved\Logs

  • AccelByte Development Toolkit (ADT) Log files: This test generates an n- number of ADT Log files, each containing test results on one level in JSON lines format. The layout of the ADT Log file is as follows:

    <header>
    ---
    <body>
    ---
    <footer>

When you use Unreal to run automated tests, the system generates an ADT Log file. The path can be anywhere and is used later by the ADT CLI to upload the file. For example:

\<ProjectRoot>\ProjectName\Saved\Logs\AutomatedTest\AutomatedTest_Windows_Level1.json

BlackBox Log files​

BlackBox Log file header​

The <header> part of the BlackBox Log File should be:

{
  "@encoding": "blackbox/snifftest+json.v2.0.0"
}

Where "@encoding" is the encoding version of this ADT Log file. Make sure the encoding value is current. This encoding tells the ADT Hub the format of this file.

BlackBox Log file body​

The <body> part of the test result should be in the following format:

{"type": "begin_test", "name": "test1"}
{"type": "msg", "msg_type": "error", "msg": "errors", "callstack":"C:\Dev\Project\Source\Project\Private\Automation\AutoCommands.cpp (777)"}
{"type": "msg", "msg_type": "warning", "msg": "warning", "callstack":"C:\Dev\Project\Source\Project\Private\Automation\AutoCommands.cpp (778)"}
{"type": "msg", "msg_type": "info", "msg": "info", "callstack":"C:\Dev\Project\Source\Project\Private\Automation\AutoCommands.cpp (779)"}
{"type": "end_test", "name": "test1", "result": "success"}

For example:

{"type":"begin_test","name":"Mantle"}
{"type":"msg","msg_type":"info","msg":"MantleFromActorsPosition test : start","callstack":"C:\Dev\Project\Source\Project\Private\Automation\AutoCommands_Locomotion.cpp (777)"}
{"type":"msg","msg_type":"info","msg":"MantleFromActorsPosition Teleport to Mantle1","callstack":""C:\Dev\Project\Source\Project\Private\Automation\AutoCommands_Locomotion.cpp (830)"}
{"type":"msg","msg_type":"info","msg":"MantleFromActorsPosition test : Mantle state reached","callstack":""C:\Dev\Project\Source\Project\Private\Automation\AutoCommands_Locomotion.cpp (877)"}
{"type":"msg","msg_type":"info","msg":"MantleFromActorsPosition from actor (Mantle1) : Mantle Success","callstack":""C:\Dev\Project\Source\Project\Private\Automation\AutoCommands_Locomotion.cpp (884)"}
{"type":"msg","msg_type":"info","msg":"MantleFromActorsPosition test : finish","callstack":""C:\Dev\Project\Source\Project\Private\Automation\AutoCommands_Locomotion.cpp (892)"}
{"type":"end_test","name":"Mantle","result":"success"}

Wrap logs inside "begin_test" and "end_test". The test fails if there are error-type logs inside.

The body of "begin_test"type consists of these parameters:

  • type of the current JSON line, in the format "type":"begin_test"
  • name of the current test in the format "name": "test1"

The body of "msg" type consists of these parameters:

  • type of the current JSON line, in the format "type":"msg"
  • msg_type is the log verbosity type (info, error, warning)
  • msg is the log message
  • callstack is the call stack where the log is called from

The body of "end_test" type consists of these parameters:

  • type of the current JSON line, in the format "type":"end_test"
  • name of the current test
  • result of the current test, "failed" if there are error log messages.

The <footer> part of the test result:

{
  test_id : "", map: "", total_test : "", failed : "", success : "", result : ""
}
  • test_id is a generated random UUID
  • map is at the map (level) where the current test is running
  • total_test is the number of total actions run in this Test (or total lines in <body> part)
  • failed is the number of failed tests
  • success is the number of successful tests
  • result is the result of the test in the current map (level). The options are success or failed. The result is success only if there are no failed results for any of the records inside the body.

Example of a test result​

Example of an ADT Log file using a game called ABShooter in Level 1:

{
  "@encoding": "blackbox/snifftest+json.v1.0.1"
}
---
{"type":"begin_test","name":"LoadMap-Level_1"}
{"type":"msg","msg_type":"warning","msg":"RegisterLocationMarkersForBodyPartCaps has been called", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"warning","msg":"GetVertexLocations returned false for mesh HeadStump.", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"info","msg":"FlushAsyncLoading: 1 QueuedPackages, 0 AsyncPackages", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"info","msg":"ULevelStreaming::RequestLevel(/Game/Maps/Game/Escape/Sections/Security/Escape_Security_VFX) is flushing async loading", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"error","msg":"USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"error","msg":"USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"end_test","name":"LoadMap-Level_1","result":"failed"}
{"type":"begin_test","name":"Flashlight-PlayerStart"}
{"type":"msg","msg_type":"info","msg":"Flashlight test : start", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"msg","msg_type":"info","msg":"Flashlight test : Success", "callstack":""C:\Dev\Project\Source\Project\Private\Automation\Something.cpp (830)"}
{"type":"end_test","name":"Flashlight-PlayerStart","result":"success"}
---
{
  test_id : "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", map: "Level_1", total_test : "2", failed : "1", success : "1", result : "failed"
}
Last updated on

On this page