Interpret automated test output
Overview
Automated tests are typically one part of an automated build pipeline, and are used to reduce the cost of a frequent build regime. Automated tests produce two kinds of output:
-
Unreal Log: This log prints in the command line if we run it there, or in the Unreal Log window if we run it in the Unreal Editor. The Unreal log file is simply the output directed to the Unreal console during the session. By default, the log is saved here:
\<ProjectRoot>\ProjectName\Saved\Logs -
AccelByte Development Toolkit (ADT) Log files: This test generates an n- number of ADT Log files, each containing test results on one level in JSON lines format. The layout of the ADT Log file is as follows:
<header>
---
<body>
---
<footer>
When you use Unreal to run automated tests, the system generates an ADT Log file. The path can be anywhere and is used later by the ADT CLI to upload the file. For example:
\<ProjectRoot>\ProjectName\Saved\Logs\AutomatedTest\AutomatedTest_Windows_Level1.json
BlackBox Log files
Each part of the file is written in JSON lines format: the header is one line, the footer is one line, and each body entry is one line. Put a line with only --- between the header and the body, and between the body and the footer. ADT reads these --- lines to find where each part starts, so keep them in the file.
BlackBox Log file header
The <header> part of the BlackBox Log File should be:
{"@encoding": "blackbox/snifftest+json.v2.1.0", "test_id": "<test id in UUID>", "map": "<map>"}
@encodingis the encoding version of this ADT Log file. Useblackbox/snifftest+json.v2.1.0. The ADT CLI checks this value when you upload the file, and ADT Web uses it to read the file.test_idis a generated random UUID that identifies this test run. Use the same value in every body line and in the footer.mapis the map (level) where the current test is running. ADT Web shows it in the Automated Test list.
BlackBox Log file body
The <body> part of the test result should be in the following format:
{"test_id": "<test id in UUID>", "type": "begin_test", "name": "test1"}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "error", "msg": "errors", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (777)", "timestamp": 1612416088}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "warning", "msg": "warning", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (778)", "timestamp": 1612416089}
{"test_id": "<test id in UUID>", "type": "msg", "msg_type": "info", "msg": "info", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands.cpp (779)", "timestamp": 1612416090}
{"test_id": "<test id in UUID>", "type": "end_test", "name": "test1", "result": "success"}
For example:
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "Mantle"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : start", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (777)", "timestamp": 1612416088}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition Teleport to Mantle1", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (830)", "timestamp": 1612416089}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : Mantle state reached", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (877)", "timestamp": 1612416090}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition from actor (Mantle1) : Mantle Success", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (884)", "timestamp": 1612416091}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "MantleFromActorsPosition test : finish", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\AutoCommands_Locomotion.cpp (892)", "timestamp": 1612416092}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "Mantle", "result": "success"}
Wrap logs inside "begin_test" and "end_test". The test fails if there are error-type logs inside.
The body of "begin_test" type consists of these parameters:
test_idof the test run, the same value as in the headertypeof the current JSON line, in the format"type":"begin_test"nameof the current test in the format"name": "test1"
The body of "msg" type consists of these parameters:
test_idof the test run, the same value as in the headertypeof the current JSON line, in the format"type":"msg"msg_typeis the log verbosity type (info, error, warning)msgis the log messagecallstackis the call stack where the log is called fromtimestampis the time the log was written, in Unix time (seconds, not milliseconds). Add it to every"msg"line. ADT Web shows it next to the log. If it is missing, the log still shows, but without a time. ADT Web lists logs in the order they appear in the file, not by timestamp.
The body of "end_test" type consists of these parameters:
test_idof the test run, the same value as in the headertypeof the current JSON line, in the format"type":"end_test"nameof the current testresultof the current test,"failed"if there are error log messages.
BlackBox Log file footer
The <footer> part of the test result:
{"test_id": "<test id in UUID>", "total_test": 0, "failed": 0, "success": 0, "result": "<result>"}
test_idof the test run, the same value as in the headertotal_testis the number of tests in the<body>part (each"begin_test"and"end_test"pair is one test). Writetotal_test,failed, andsuccessas JSON numbers (for example,2), not as strings (for example,"2"). The0values in the template above are placeholders.failedis the number of failed testssuccessis the number of successful testsresultis the overall result of the test run. The options aresuccessorfailed. The result issuccessonly if there are no failed results for any of the records inside the body.
Example of a test result
Example of an ADT Log file using a game called ABShooter in Level 1:
{"@encoding": "blackbox/snifftest+json.v2.1.0", "test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "map": "Level_1"}
---
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "LoadMap-Level_1"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "warning", "msg": "RegisterLocationMarkersForBodyPartCaps has been called", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416088}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "warning", "msg": "GetVertexLocations returned false for mesh HeadStump.", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416089}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "FlushAsyncLoading: 1 QueuedPackages, 0 AsyncPackages", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416090}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "ULevelStreaming::RequestLevel(/Game/Maps/Game/Escape/Sections/Security/Escape_Security_VFX) is flushing async loading", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416091}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "error", "msg": "USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416092}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "error", "msg": "USoundControllerComponent::BeginPlay - sound slots not configured!", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416093}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "LoadMap-Level_1", "result": "failed"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "begin_test", "name": "Flashlight-PlayerStart"}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "Flashlight test : start", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416094}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "msg", "msg_type": "info", "msg": "Flashlight test : Success", "callstack": "C:\\Dev\\Project\\Source\\Project\\Private\\Automation\\Something.cpp (830)", "timestamp": 1612416095}
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "type": "end_test", "name": "Flashlight-PlayerStart", "result": "success"}
---
{"test_id": "fc341e58-9a5a-48e6-9ffc-c3b398f935e5", "total_test": 2, "failed": 1, "success": 1, "result": "failed"}