1 # Copyright (c) 2016 DENX Software Engineering GmbH
2 # Heiko Schocher <hs@denx.de>
4 # SPDX-License-Identifier: GPL-2.0+
10 tbot is a tool for executing testcases on boards.
11 Source code found on [1]
18 https://github.com/hsdenx/tbot/blob/master/doc/tbot_structure.png )
20 - Virtual laboratory (VL)
21 VL is the basic environment that groups:
22 - [a number of] boards - target devices on which tbot executes testcases.
26 A piece of python code, which uses the tbot class from [1].
27 Tbot provides functions for sending shell commands and parsing the
28 shell commands output.
29 Tbot waits endless for a shell commands end (detected through reading
31 A TC can also call other TC-es.
34 Tbot not really waits endless, for a shell commands end, instead
35 tbot starts a watchdog in the background, and if it triggers, tbot
36 ends the TC as failed. In the tbot beginning there was a lot of
37 timeouts / retry cases, but it turned out, that waiting endless
38 is robust and easy ...
40 - Host PC (where tbot runs, currently only linux host tested)
41 must not a powerful machine (For example [3], I use a
42 raspberry pi for running tbot and buildbot)
45 - Host PC connects through ssh to the Lab PC
46 -> so it is possible to test boards, which
47 are not at the same place as the Host PC.
48 (Lab PC and Host PC can be the same of course)
49 -> maybe we can setup a Testsystem, which does nightly
50 U-Boot/Linux builds and test from current mainline U-Boot
51 on boards wherever they are accessible.
53 - necessary tasks a Lab PC must deliver:
54 - connect to boards console through a shell command.
55 - power on/off boards through a shell command
56 - detect the current power state of a board through
60 - tftp server (for example loading images)
61 - nfs server (used as rootfs for linux kernels)
62 - Internet access for example for downloading
63 U-Boot source with git.
64 - toolchains installed for compiling source code
66 -> a linux machine is preffered.
68 - currently only Lab PC with an installed linux supported/tested.
71 the boards on which shell commands are executed.
74 equals to the software, the board is currently running.
76 Currently tbot supports 2 board states:
77 - "u-boot", if the board is running U-Boot
78 - "linux", if the board is running a linux kernel
80 It should be easy to add other board states to tbot, see
81 https://github.com/hsdenx/tbot/tree/master/src/lab_api/state_[u-boot/linux].py
83 A board state is detected through analysing the boards
84 shell prompt. In linux, tbot sets a special tbot prompt,
85 in U-Boot the prompt is static, and configurable in tbot through
88 A TC can say in which board state it want to send shell commands.
89 Tbot tries to detect the current board state, if board is not in
90 the requested board state, tbot tries to switch into the correct
91 state. If this fails, the TC fails.
93 It is possible to switch in a single TC between board states.
96 tbot creates while executing testcases so called events.
97 After tbot ended with the testcase it can call event_backends,
98 which convert the events to different formats. more info:
100 https://github.com/hsdenx/tbot/blob/master/doc/README.event
102 demo for a event backend:
103 http://xeidos.ddns.net/tests/test_db_auslesen.php
105 - tbot cmdline parameters:
107 $ python2.7 src/common/tbot.py --help
108 Usage: tbot.py [options]
111 -h, --help show this help message and exit
112 -c CFGFILE, --cfgfile=CFGFILE
113 the tbot common configfilename
114 -l LOGFILE, --logfile=LOGFILE
115 the tbot logfilename, if default, tbot creates a
117 -t TC, --testcase=TC the testcase which should be run
118 -v, --verbose be verbose, print all read/write to stdout
119 -w WORKDIR, --workdir=WORKDIR
120 set workdir, default os.getcwd()
123 tbot needs the following files for proper execution:
125 - tbot board configuration file (option -c):
126 A board configuration file contains settings tbot needs to
127 connect to the Lab PC and board specific variable settings
130 - name of the logfile tbot creates (option -l)
131 defaultname: 'log/' + now.strftime("%Y-%m-%d-%H-%M") + '.log'
133 - tbots working directory (option -w)
135 - the testcasename tbot executes (option -t)
137 You are interested and want to use tbot?
138 If so, please read on the file:
139 tools/tbot/README.install
143 Heiko Schocher <hs@denx.de>
147 [1] https://github.com/hsdenx/tbot
148 [2] http://www.denx.de/wiki/DUTS/DUTSDocs
149 [3] automated Testsetup with buildbot and tbot doing cyclic tests
150 (buildbot used for starting tbot TC and web presentation of the
151 results, all testing done through tbot):
152 http://xeidos.ddns.net/buildbot/tgrid
153 Host PC in Letkes/hungary
156 Fancy things are done here, for example:
157 - http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/43/steps/shell/logs/tbotlog
158 (I try to cleanup the logfile soon, so it is not so filled with crap ;-)
159 A first step see here:
160 http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/45/steps/shell/logs/tbotlog
161 (same TC now with the new loglevel = 'CON' ... not yet perfect)
165 - get a list of patchwork patches from my U-Boots ToDo list
166 - download all of them, and check them with checkpatch
167 and apply them to u-boot.git
168 - compile U-Boot for the smartweb board
169 - install the resulting images on the smartweb board
172 - more TC should be added here for testing U-Boot
174 - automatic "git bisect"
175 https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
176 http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
178 If a current U-Boot image not works on the tqm5200 board
179 this TC can be started. It starts a "git bisect" session,
180 and compiles for each step U-Boot, install it on the tqm5200
181 board, and tests if U-Boot works !
183 At the end, it detects the commit, which breaks the board
185 This TC is not dependend on U-Boot nor on a special board. It
186 needs only 3 variables:
187 tb.board_git_bisect_get_source_tc: TC which gets the source tree, in which
188 "git bisect" should be executed
189 tb.board_git_bisect_call_tc: TC which gets called every "git bisect" step,
190 which executes commands for detecting if current source code is OK or not.
191 This could be a TC which compiles U-Boot, install it on the board and
192 executes TC on the new booted U-Boot image. ! Board maybe gets borken,
193 as not all U-Boot images work, so you must have a TC which install U-Boot
194 image for example through a debugger.
195 tb.board_git_bisect_good_commit: last nown good commit id