Old stuff, very old

Test design

Our test system has a pretty simple configuration, so I do not want to bother you with the detailed description of it. Simply - it's just a standalone execution of a test.py file with all requirements (test setup and tear down) in local internal test utility. Helper local utility has a usual amount of preparation functions, test data loading and some other configurations. In the very end of test loading, it just compares a node-set result of "real" scan (software and hardware nodes in a database after the scan) with predefined node-set (from test.py). Node set we usually expect at the end of the test should be added by the developer when product discovery pattern is ready or have a new feature or fix.

Now I'm trying to implement entirely new and somewhere different test design because current is not flexible enough and require a lot of time to run. I'll describe the original design of test in future, in a separate thread.

Making tests was torture when I came to "named" project. Nobody understands or even can describe the whole picture of a test run. People make tests by some "golden standard" which never works and usually was un-reproduce-able. In other words - tests do not work (and never worked before for this particular case). Team members were trying to automate some routines of test writing or executing from time to time, but with no big luck. I was trying to make that one "gold standard" until I tired of this.

First blood

The first attempt was to implement a better syntax highlighting for TPL (TPL - tideway pattern language, further "tpl" and "tplpre"). We used very poor syntax highlight for N++ and Vim from the beginning. This type of editors is good enough for small changes and bash scripts, but not for the development and dynamical modifications for tpl. Had begun from Sublime text 2, but later I moved on 3rd version mainly. Here is the latest version on GitHub. Further development of it was suspended and transferred to Atom. 

Naive usage:

Forgot to mention - I've tried to automate pattern loading, syntax checking and test run on N++, firstly, before I start developing syntax highlight. And the very first attempt was just to automate small amount of daily routine using windows CMD + OpenSSH and N++ build system. Fortunately, I  can’t find that old records to show something for you now.

Syntax, completions, tests small automation attempts.

Hence with new syntax highlight and auto-completion snippets with TPLlinter module tplint (by Ladkau, Matthias), the development routine became smarter and more enjoyable. New syntax markup allows you to close code blocks, insert smart indent, close most common tags, and keywords and even show and complete all tpl built-in functions from official documentation.

Because I finish implementing fixture for most disrupting part of the development process I have time to rethink the whole development cycle and see other weak places. Happily, Sublime had a built-in Python interpreter and build-system. I've started my research and implementation of small automation with adding most common commands for ADDM via Sublime build system with python logic and scripts (start\restart tests, load pattern, start a scan, etc.)

Because I finish implementing fixture for most disrupting part of the development process I have time to rethink the whole development cycle and see other weak places. Happily, Sublime had a built-in Python interpreter and build-system. I've started my research and implementation of small automation with adding most common commands for ADDM via Sublime build system with python logic and scripts (start\restart tests, load pattern, start a scan, etc.)

Output example:

INFO: Working directory is:                                     D:\Doc\PerForce\addm\tkn_main\tku_patterns\CORE\BMCRemedyARSystem
INFO: Scan will be named as:                                    BMCRemedyARSystem
INFO: Syntax:                                                   PASSED!
INFO: TPL version:                                              11.0
INFO: TPL tpl_folder:                                           tpl110
INFO: TPLPreprocessor run all on all files in directory:        BMCRemedyARSystem
INFO: TPL_Preprocessor:                                         PASSED!
INFO: ADDM ip is:                                               192.168.5.8
INFO: Open session with ADDM.
INFO: Making zip of all .tpl in folder:                         D:\Doc\PerForce\addm\tkn_main\tku_patterns\CORE\BMCRemedyARSystem\tpl110\
INFO: Upload zip to ADDM folder:                                /usr/tideway/TKU/Tpl_DEV/
INFO: zipFilename:                                              test_pattern.zip
INFO: Adding pattern:                                           BMCRemedyARSystem.tpl
INFO: Adding pattern:                                           CDM_Mapping.tpl
INFO: Adding pattern:                                           Common_Functions.tpl
INFO: Adding pattern:                                           DiscoveryFunctions.tpl
INFO: Adding pattern:                                           RDBMSFunctions.tpl
INFO: Adding pattern:                                           RDBMS_Functions.tpl
INFO: Adding pattern:                                           SearchFunctions.tpl
INFO: zip_path:                                                 D:\Doc\PerForce\addm\tkn_main\tku_patterns\CORE\BMCRemedyARSystem\tpl110\test_pattern.zip
INFO: Patterns zip to ADDM upload:                              PASSED!
======================================================================================================================================================
INFO: Installing and activating pattern modules. Result:
Uploaded /usr/tideway/TKU/Tpl_DEV/test_pattern.zip as knowledge upload "test_pattern upload 20"
1 knowledge upload activated
INFO: Upload activating:                                        PASSED!
INFO: Discovery mode is:                                        SET PLAYBACK MODE
Discovery status:                                               Discovery is RUNNING
======================================================================================================================================================
INFO: Discovery started
INFO: Host(s) to scan:                                          0.0.0.0
INFO: Scan will be named as:                                    BMCRemedyARSystem
INFO: Scan has been added to the Currently Processing Runs:
Added snapshot scan range
ID                               Enabled Label             User   Scan type Range 
366928346894adcc79627f000001099a True    BMCRemedyARSystem system Snapshot  0.0.0.0
[Finished in 20.0s]

Some magic

Functions and routines implemented:

In general, all current functions list is present TPL IDE on GitHub:

• [X] syntax tests (require 3rd party module tplint by Ladkau, Matthias)
• [X] importing linked patterns (working only if the dev environment found):
  • [X] from a current pattern;
  • [X] from a current pattern and each imported;
  • [X] from a current pattern and each imported and test.py;
• [X] Run TPLPreprocessor:
  • [X] on the existing pattern file;
  • [X] on imported pattern files;
• [X] SSH to ADDM for options:
  • [X] check tpl version;
  • [X] check DEV paths;
  • [X] pattern uploading;
  • [X] auto scan start;
• [X] Start scan;
• [X] Tests
  • [X] Run pattern test;
  • [X] Run pattern related tests;

Use case

Sublime and later Atom plugins for syntax and simple automation allow a developer to add some most common scenarios in keyboard shortcuts. The cycle was like:

• Make a draft pattern (early version with raw commands captions)
  • Check syntax, execute a preprocess (internal tool)
  • Run discovery with the predefined host where product installed
  • Get results, change a pattern to catch new outputs.
• Repeat

Also, this version includes separate logic for external developers, who work on custom tpl patterns outside of BMC. It detects paths hierarchy and extra arguments to use other steps to automate routines for other developers.

Raw examples

Almost the whole cycle was automated now. Now you can watch logs and makes changes in pattern body without any standard routines. System utility imports every required pattern, install it into Discovery Knowledge, run the scan and show a limited output. Besides that, I even wanted to add results parsing and querying, but that day never came. I moved forward to make own test framework "Octopus." More cases you can find on GitHub with following links:

• https://github.com/trianglesis/BMC_TPL_IDE
• https://github.com/trianglesis/language-tplpre
• https://github.com/trianglesis/bmc_tpl

Furthermore:

Plugin development was finished after manager’s request to use an open source Atom editor. Moving process requires a lot of changes to run from Sublime to Atom. I can not port it with simple files format converting to a new style. As a result, I've changed a lot in current design and fix old bugs of syntax highlight, build system. Moreover, during a move, I used a new format of auto-completion logic from Atom built-in. New way also allows me to include references for any documented tpl functions and keywords. Automation logic I also change, it became friendlier to the user, and also it contains additional output messages and error handling. The last and most complex feature was the test running automation.

Initial test automation attempts:

Then, tests automation procedure on that level allows a user to select the desired test level. And then test single case for a pattern, or include all "patterns" which were imported and changed from last time. More advanced logic also consists of a routine to execute all related tests. Which means you can perform current pattern test and ALL other tests where this pattern was included somehow or imported. Probably, products like RDBMS systems can run for hours.

And I also want to mention, that last TPL IDE version can execute tests in a development environment with configured shares and file hierarchy and single mode, when just only selected files the to a temporary folder. 

The core and the beginning

Finally, an example and the core code which define the era of Octopus for our project is situated here: bmc_tplpre/check_ide/upload.py. Small function "tests_executor" is the heart of all further implementations and framework development. Everything was just a preparation for this crucial moment =)

pre_cmd = ". ~/.bash_profile;"
wd_cmd = "cd "+test['rem_test_wd']+";"
cmd_test = "/usr/tideway/bin/python -u "+test['rem_test_path']+test_args
cmd = pre_cmd + wd_cmd + cmd_test

Note: this is a very early example, and I stopped the development of this particular plugin. Now everything became more complicated, but anyway, this is the place where it all has begun.

• PREVIOUS: Test automation or: How I Learned to Stop Worrying and Love Async __init__
• NEXT: How first automation works with examples.