ATRT documentation

ATRT_SETUP_README.txt

Author: Serge Kozlov, MySQL

Date: 03/23/2006

Contents

1. Overview

2. Setup

3. Preparing for testing

4. CPCD. Command line and configuration options

5. d.tmp. Examples

6. Test Case File format. Examples

Note: how to run ATRT tests please look ATRT_USAGE_README.txt

  1.Overview

  ==========

  Auto Test Run Test (ATRT) is a home made test frame work. This frame work can 

start and stop processes on different Linux hosts through another executable 

called Cluster Process Control Daemon (CPCD) running on the other hosts. 

It uses for testing cluster configurations located on different machines. 

ATRT isn't replacement for mysqltest. In fact, ATRT can invoke mysql-test as 

samples blow show. This framework has been designed to run most any test 

(mysqltest testcases, stress tests, any applications) in cluster and detects 

all errors and issues which happened in any node. In addition, ATRT starts 

applications that are defined as test with arguments and will analyze log files 

from all nodes that are produced.

  2.Setup

  =======

  Following steps described how to setup ATRT framework on a cluster:

* You need to have cloned source tree (e.g mysql-5.1-new) on machine where you 

  plan to use ATRT.

* Compile and install build (e.g. for Linux/x86 can use 

  BUILD/compile-pentium-max --prefix=/path/to/installation).

* Copy $MYSQL_DIR on all machines and on same path which you plan to use as 

  nodes (you can use scp utility or ask JonathanMiller about distribution on 

  ndbXX servers).

* Compile and install ATRT:

  * Go to $TREE_ROOT/storage/ndb/test

  * do make

  * do make install

  * Make sure that $MYSQL_DIR/bin now contains files like that: testBasic, 

    testBlobs, testDict and so on

  * Make sure that $MYSQL_DIR/mysql-test/ndb now contains files like that: atrt,

    atrt-analyze-result.sh, atrt-setup.sh, make-config.sh and so on

  * Add $MYSQL_DIR/mysql-test/ndb to PATH and make sure that you can call atrt 

    from any place on disk.

* Configure CPCD processes on all machines. Repeat following steps for each 

  node:

  * Create $VAR_DIR/run/ndb_cpcd directory

  * Create /etc/ndb_cpcd.cnf file. It will configuration file ndb_cpcd daemon. 

    It strongly recommended instead command line options because more simply. 

    More option for configuration can fe found in this document in section CPCD

  * Add to file following text:

	[ndb_cpcd]

	work-dir= $VAR_DIR/run/ndb_cpcd # e.g. /mysql/builds/5.1/var/run/ndb_cpcd

	logfile= $VAR_DIR/run/ndb_cpcd/log.txt # e.g. /mysql/builds/5.1/var/run/ndb_cpcd/log.txt

	debug= 1

	user= ndbdev

  * Register ndb_cpcd in etc/initab. Hint: you can use more simply way if use 

    Linux - add following line to /etc/rc.d/rc.local file

	$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &

  * Start ndb_cpcd

	$MYSQL_DIR/libexec/ndb_cpcd > /dev/null 2>&1 &

  * Open log file and make sure that ndb_cpcd process started properly. Ususal mistakes are: wrong definition of paths and ndb_cpcd process already running. You can test ndb_cpcd from any ndb system by using the command line below. If nothing is returned the process is up and running. Otherwise you will see Failed to connect to node:1234:

	$MYSQL_DIR/libexec/ndb_cpcc node

Now ATRT Setup done. After accomplishment all steps above you should be have 

following:

* Each node contains fresh build

* Each node has configured and started ndb_cpcd process.

* The node that will used for starting ATRT has installed atrt binaries/scripts

  and some tests.

  3.Preparing for testing

  =======================

* Create new empty directory where you plan to use for testing. Note: if you 

  plan to test different cluster configuration you need to create own directory 

  for each cluster configuration. Do not use same directory.

* Create d.tmp file for your cluster configuration. See details in this 

  document in section d.tmp.

* Run make-config.sh ./d.tmp. It will create necessary directories and files for 

  ATRT.

* Create ATRT testcase file(s) (e.g. test1.atrt). See details in this document 

  in section Test Case File format. Examples.

* Put your test application into $MYSQL_DIR/bin directory. If your application 

  requires another directory (e.g. mysql-test-run.pl) create redirect script 

  such as example below:

	#!/bin/sh

	set  -x

	cd $MYSQL_BASE_DIR/mysql-test

	./mysql-test-run.pl --with-ndbcluster --ndb-connectstring=$*

Now preparing for testing done.

  4.CPCD.

  =======

  The CPCD process needs to be running on each host contacts CPCD to tell it 

what process to execute. In other word for testing each node should have to 

running ndb_cpcd. Usually ndb_cpcd can be found in storage/ndb/src/cw/cpcd for 

source tree or in libexec/ndb_cpcd for binary distribution. ndb_cpcd uses 

configuration file /etc/ndb_cpcd.cnf but all options from one can be replaced 

by command line. Names of options in command line are same as from 

configuration file except some some commands have short notation (-X).

ndb_cpcd command line options

-w, --work-dir=name

    Work directory. Should be exist before starting ndb_cpcd. Usually it's 

    $VARDIR/run/ndb_cpcd

-p, --port=#

    TCP port to listen on. By default 1234

-S, --syslog

    Log events to syslog

-L, --logfile=name

    File to log events to. Usually it's $VARDIR/run/ndb_cpcd/logfile.txt

-D, --debug

    Enable debug mode.

-u, --user=name

    Run as user

  5.d.tmp

  =======

  The d.tmp file is used to create the d.txt file (configuration file for ATRT) 

and the config.ini file (configuration file for MySQL Cluster). This file does 

not have to be called d.tmp, it can be called by any name. The file will be feed 

to make-config.sh. All options in the file separated by two parts:

* Original options. They are located before '-- cluster config'. Description of 

all these options are below.

* Options for config.ini. They are located after '-- cluster config'. All 

options, sections and values completely coincide with config.ini.

d.tmp options

baseport

    Port used for communicating to the cluster on.

basedir

    basedir has to point to the root of the mysql install. Note that ATRT will 

    create a run directory under the base directory. All test directories and 

    files created will be copied to all hosts in the test under the basedir/run directory.

mgm

    Host to NDB Cluster manager on. Put hostnames separated by blanks.

ndb

    Host(s) to run NDB Data Nodes on. Put hostnames separated by blanks.

api

    Host(s) that NDB API should be ran on. Put hostnames separated by blanks.

mysqld

    Host(s) that mysqld processes should be started on. Put hostnames separated by blanks.

mysql

    Host(s) that mysql processes should be started on. Put hostnames separated 

    by blanks.

Example d.tmp for cluster configuration: 1 ndb node, 1 replica, 1 mgm, 1 api, 

1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17

baseport: 14000

basedir: /home/ndbdev/skozlov/builds

mgm: ndb16

ndb: ndb17

api: ndb16

mysqld ndb16

mysql ndb16

-- cluster config

[DB DEFAULT]

NoOfReplicas: 1

[MGM DEFAULT]

PortNumber: 14000

ArbitrationRank: 1

Example d.tmp for cluster configuration: 2 ndb nodes, 2 replicas, 1 mgm, 1 api, 

1 mysql, 1 mysqld. Available hosts for nodes: ndb16, ndb17, ndb18

baseport: 14000

basedir: /home/ndbdev/skozlov/builds

mgm: ndb16

ndb: ndb17 ndb18

api: ndb16

mysqld ndb16

mysql ndb16

-- cluster config

[DB DEFAULT]

NoOfReplicas: 2

[MGM DEFAULT]

PortNumber: 14000

ArbitrationRank: 1

Example d.tmp for cluster configuration: 4 ndb nodes, 4 replicas, 1 mgm, 3 api, 

2 mysql, 1 mysqld. Available hosts for nodes: ndb14, ndb15, ndb16, ndb17, ndb18

baseport: 14000

basedir: /home/ndbdev/skozlov/builds

mgm: ndb16

ndb: ndb17 ndb18 ndb15 ndb14

api: ndb16 ndb17 ndb18

mysqld ndb16

mysql ndb16 ndb17

-- cluster config

[DB DEFAULT]

NoOfReplicas: 4

[MGM DEFAULT]

PortNumber: 14000

ArbitrationRank: 1

  6.Test Case File Format. Examples.

  ==================================

  A test file consists of a list of test cases. Each test case is ended by an 

empty/blank line. Each test case is described by a set of name value pairs. 

ATRT looks for tests executables, shell and Perl scripts in the basedir/bin 

directory.

Test Case File options:

max-time

    This value is in seconds. Test ATRT how long to allow the test to run before 

    marking the test a failure and moving on to the next test. If the test 

   finishes before max-time, it will move on to the next test.

cmd

    Tells ATRT which test or script to run from the bin directory

args

    These are the command-line arguments to be passed to the test or script 

    that is being executed.

type

    Currently only bench. See ATRT Command-line parameters for details.

run-all

    will start the same command for each defined api/mysql (normally it only 

    started in 1 instance)

Example 1.

ATRT starts test $MYSQL_DIR/bin/testBlobs without arguments and sets time for 

execution as 10 min. testBlobs is binary application

max-time: 600

cmd: testBlobs

args:

Example 2.

ATRT starts test $MYSQL_DIR/bin/testRead -n PkRead and sets time for execution 

as 20 min. testRead is binary application

max-time: 1200

cmd: testRead

args: -n PkRead

Example 3.

ATRT starts test $MYSQL_DIR/bin/atrt-mysql-test-run -force and sets time for 

execution as one hour. atrt-mysql-test-run is bash script and it points to 

$MYSQL_DIR/mysql-test/mysql-test-run. In fact this test will start 

mysql-test-run --force that means the execution all mysqltest testcases in 

mysql-test/t directory.

max-time: 3600

cmd: atrt-mysql-test-run 

args: --force

Example 4.

ATRT starts test $MYSQL_DIR/bin/MyTest1 -n and sets time for execution as 2 min.

Then starts $MYSQL_DIR/bin/MyTest2 for each defined mysql/api node and set 

timeout 3 min.

max-time: 120

cmd: MyTest1 

args: -n

max-time: 180

cmd: MyTest2 

run-all: yes

ATRT_USAGE_README.txt

Author: Serge Kozlov, MySQL

Date: 03/23/2006

Contents

1. How to run

2. Results

3. ATRT Command Line Options

Note: how to setup ATRT please look ATRT_SETUP_README.txt

  1.How to run.

  =============

Simple way to start atrt:

	atrt --testcase-file=atrt_test_case_file

Command line above doesn't produce any log files. Better use following :

	atrt --log-file=log.txt --testcase-file=atrt_test_case_file

Now we can look log.txt for investigation any issues. If it is insufficiently

then add one or more -v arguments:

	atrt -v -v --log-file=log.txt --testcase-file=atrt_test_case_file

If the test case file contains two or more test we can add -r options for 

preventing stopping testing if one test fails (like --force for mysql-test-run)

	atrt -v -v -r --log-file=log.txt --testcase-file=atrt_test_case_file

The line below is optimal solution for testing:

	atrt -v -v -r -R --log-file=log.txt --testcase-file=atrt_test_case_file

All additional command line arguments and description of used in exampes above 

available in this document in section atrt command line options

  2.Results.

  ==========

  Unlike mysql-test-run frame work atrt doesn't inform to console passed/failed 

status of tests. You need to use --log-file option and look into log file for 

getting information about status of tests. When atrt finished you can look into 

log file defined --log-file option. It's main source of information about how 

were performed atrt tests. Below added the examples of content of log-file for 

different failures (except example 1 for passed test). Examples include 

probable cases with reasons and recommended solutions and cover not test issues 

only but mistakes of atrt configuration or cluster settings. As ATRT testcase 

file used simple test included in MySQL installation:

	max-time: 600

	cmd: testBasic

	args: -n PkRead? T1

Of course these examples don't cover all possible failures but at least most 

probable and often appearing.

Note: Before start atrt I recommend try to run selected cluster configuration 

by manual and make sure that it can work: enough to run mgmd, ndbd, mysqld and 

look via mgm to status of these nodes

Example 1. Test passed

2006-03-02 15:36:51 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 15:36:51 [ndb_atrt] INFO     -- Starting...

2006-03-02 15:36:51 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 15:36:51 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 15:36:55 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 15:37:11 [ndb_atrt] INFO     -- Ndb start completed

2006-03-02 15:37:11 [ndb_atrt] INFO     -- #1 - testBasic -n PkRead? T1

2006-03-02 16:37:16 [ndb_atrt] INFO     -- #1 OK (0)

Example 2. Test failed.

Reason: ATRT not started properly. d.txt not found. Probably necessary folders 

and d.txt file don't exist.

Solution: run make-config.sh d.tmp and try again.

2006-03-02 18:32:08 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:32:08 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:32:08 [ndb_atrt] CRITICAL -- Failed to open process config file: d.txt

Example 3. Test failed.

Reason: ATRT not started properly. Necessary folders were removed but d.txt file 

exists.

Solution: run make-config.sh d.tmp and try again.

2006-03-02 18:30:54 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:30:54 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:30:54 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 18:30:54 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 18:30:58 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 18:31:00 [ndb_atrt] CRITICAL -- Failed to setup process

Example 4. Test failed.

Reason: ATRT not started properly. node2 hasn't running ndb_cpcd process.

Solution: log into node2 and starts ndb_cpcd process.

2006-03-02 18:15:05 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:15:05 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:15:05 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 18:15:05 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 18:15:05 [ndb_atrt] ERROR    -- Unable to connect to cpc node2:1234

Example 5. Test failed.

Reason: ATRT not started properly. baseport option isn't defined in d.tmp or 

baseport and PortNumber are different.

Solution: correct d.tmp, run make-config.sh d.tmp and try again.

2006-03-02 18:25:31 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:25:31 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:25:31 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 18:25:31 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 18:25:36 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 18:26:08 [ndb_atrt] CRITICAL -- Unable to connect to ndb mgm node1:0

Example 6. Test failed.

Reason: ATRT not started properly. basedir option points to wrong path.

Solution: correct basedir, run make-config.sh d.tmp and try again.

2006-03-02 18:40:10 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:40:10 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:40:10 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 18:40:10 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 18:40:14 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 18:40:16 [ndb_atrt] ERROR    -- Unable to start process: Failed to start

Example 7. Test failed.

Reason: ndb nodes have problems on starting.

Solution: Check configuration of ndb nodes.

2006-03-02 18:46:44 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 18:46:44 [ndb_atrt] INFO     -- Starting...

2006-03-02 18:46:44 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 18:46:44 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 18:46:50 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 18:52:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4

2006-03-02 18:58:59 [ndb_atrt] CRITICAL -- wait ndb timed out 3 3 4

Example 8. Test failed.

Reason: test application not found.

Solution: Correct file name in ATRT test case file and make sure that the file 

exists in $MYSQL_DIR/bin directory.

2006-03-02 20:21:54 [ndb_atrt] INFO     -- Starting...

2006-03-02 20:21:54 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 20:21:54 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 20:21:57 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 20:22:12 [ndb_atrt] INFO     -- Ndb start completed

2006-03-02 20:22:12 [ndb_atrt] INFO     -- #1 - testBasic123 -n PkRead? T2

2006-03-02 20:22:13 [ndb_atrt] ERROR    -- Unable to start process: Failed to start

Example 9. Test failed.

Probable reasons:

* wrong arguments for test application

* itself test failed

* timeout reached

Solution: Since mgmd/ndbd nodes started properly in such case then try to 

investigate log files in result/X.api/, result/X.mysqld, result/X.mysql 

directories.

2006-03-02 19:59:35 [ndb_atrt] INFO     -- Setup path not specified, using /home/ndbdev/skozlov/asetup

2006-03-02 19:59:35 [ndb_atrt] INFO     -- Starting...

2006-03-02 19:59:35 [ndb_atrt] INFO     -- Connecting to hosts

2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node1:1234

2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node2:1234

2006-03-02 19:59:35 [ndb_atrt] DEBUG    -- Connected to node3:1234

2006-03-02 20:00:49 [ndb_atrt] INFO     -- (Re)starting ndb processes

2006-03-02 20:01:05 [ndb_atrt] INFO     -- Ndb start completed

2006-03-02 20:01:05 [ndb_atrt] INFO     -- #1 - testBasic -n PkRead? T1

2006-03-02 20:01:38 [ndb_atrt] INFO     -- #1 FAILED(256)

  3.ATRT Command Line Options.

  ============================

--process-config=string

    Specify ATRT configuration file. If not specified, ATRT will look in local 

    directory for the d.txt file.

--setup-path=string

    This path points to place where necessary directories (created by 

    make-config) can be found. Note: d.txt should be in same directory where 

    you start atrt!

-v (verbose)

    * without the option: atrt prints only [CRITICAL] events

    * one -v: atrt prints [CRITICAL], [INFO] events

    * two -v: atrt prints [CRITICAL], [INFO], [DEBUG] events

--log-file=string

    Used to specify file to log ATRT's results on starting application and 

    running tests.

--testcase-file=string -f

    Used to feed ATRT test cases in a text file.

--report-file=string

    File to record test results

-i, --interactive

    ATRT terminates on first test failure

-r, --regression

    Continues even on test failures

-b, --bench

    Always produce report