Merge UpCodes improvements (#87)

* Group tests by model with the `GroupByModel` param
* Tests in same fixture with same model path will run with the same model (without reopening the model)
* Additional fixes:
    - Remember the directory for file pickers
    - Automatically set the working/results directory
    - Automatically infer the version of Revit to use from the assembly (based on the RevitAPI linked against)
    - Clean up a bunch of stale variables, update parts to modern C# standards
    - Fix by category grouping to show tests that aren't in a category

* Nuget package definition

* Fix the test runner such that it spits out an error when a text model is not found

* Various UI improvements:
* Show how many tests pass/fail after a test run
* Auto expand tree with failing tests

* fixup! Merge pull request #4 from marchello2000/mark/failing_tests_indicator

* Clarity improvements:
* Add `ERROR` and `WARNING` to appropriate WriteLines to clearly highlight when
  something goes wrong
* Compact the UI a bit and give more room to console logger
* Clearly state how many tests pass/fail/skip in the last run in the console log for the gui

* Nuget version update

* Add ability to marshal console logs from unit test

* Spit out information to console as the test complete

* Add ability to clear in the console box

* Update nuspec version and readme

* Add missing empty.rfa file to the nuget package

* Porting fix for #71

* Properly fail tests whose model isn't found

* Fix null deref in console mode

* Add ability to specify wildcard for a model file name
The way it's implemented is as follows:
1. During assembly load, check if a test has a wildcard for the model, e.g.
```
[TestModel(@"C:\RevitModels\test_models\*.rvt")]
```
or
```
[TestModel(@"C:\RevitModels\test_models\*.rfa")]
```

2. Enumerate all files that match the wildcard in the specified directory
3. For each model file found, create a separate entry for the current test with this model

* Add proper error handling for wildcard models
Add error when no model files are found with a given wildcard

* Initialize output redirector earlier to catch command line arguments

* Add a timeout to Revit termination
In case Revit doesn't terminate within a timeframe (45s) it is force terminated
after test run is complete (only works for -continuous right now)

* Update README.md

* Automated build with AppVeyor
* Use Nugets for Revit dependencies (e.g. RevitAPI.dll and RevitAddingUtility.dll)
* Remove platform architecture warnings
* Remove some unused variables and other warnings from build
* Remove double definition of NUnitResultTypes
* Add ability to override Revit installation enumeration during UnitTests for CI machines with Revit
* Update Nuget package and add AppVeyor.yaml file

* Fix automatic nuget publishing

* Add instructions on how to run RTF in CI

* Pull request fixups
Revert to ReflectionOnlyLoadFrom for assembly loader because Dynamo relies on it

* PR Feedback:
* Ensure all new methods/classes have XML docs
* Ensure all memebers have access modifiers

* Update readme

* PR Feedback: do not distribute nunit/prism/etc libraries
Using Costura to merge the needed binaries into the RTF assemblies
such that they don't need to be redistributed without correct nuget usage

* Adding license info for used dlls

Author: marchello2000

.gitignore

@@ -49,6 +49,7 @@ x64/
 build/
 [Bb]in/
 [Oo]bj/
+.vs/
 
 # MSTest test Results
 [Tt]est[Rr]esult*/
@@ -130,7 +131,7 @@ publish/
 
 # NuGet Packages Directory
 ## TODO: If you have NuGet Package Restore enabled, uncomment the next line
-#packages/
+packages/
 
 # Windows Azure Build Output
 csx

README.md

@@ -1,13 +1,25 @@
-## RevitTestFramework
+# RevitTestFramework
 
-The Revit Test Framework (RTF) allows you to conduct remote unit testing on Revit. RTF takes care of creating a journal file for running revit which can specify a model to start Revit, and a specific test or fixture of tests to Run. You can even specify a model to open before testing and RTF will do that as well. 
+The Revit Test Framework (RTF) allows you to conduct remote unit testing on Revit. RTF takes care of creating a journal file for running revit which can specify a model to start Revit, and a specific test or fixture of tests to run. You can even specify a model to open before testing and RTF will do that as well. 
+
+[![Build status](https://ci.appveyor.com/api/projects/status/oqd23280efmaimmm/branch/mark/Revit2019?svg=true)](https://ci.appveyor.com/project/marchello2000/revittestframework/branch/mark/Revit2019) 
+[![NuGet](https://img.shields.io/nuget/v/revittestframework.svg)](https://www.nuget.org/packages/revittestframework)
 
 ## Applications
 
-##### RevitTestFrameworkConsole.exe  
+There are two ways to run RTF:
+* **RevitTestFrameworkConsole.exe**  
+   A Console application to run test from a command line
+* **RevitTestFrameworkGUI.exe**  
+   A GUI application that allows easy selection of tests to run and check results
+
+Both options run the tests in the same way
+
+### RevitTestFrameworkConsole.exe  
 A console application which allows running RTF without a user interface. If you'd like to learn more about the command line options for RTF, you can simply type "RevitTestFrameworkConsole -h" and you'll get something like this:
 ```
-  Options:   
+Options:   
+
          --dir=[VALUE]          The full path to the working directory. The working directory is the directory in which RTF will generate the journal and the addin to Run Revit. Revit's run-by-journal capability requires that all addins which need to be loaded are in the same directory as the journal file. So, if you're testing other addins on top of Revit using RTF, you'll need to put those addins in whatever directory you specify as the working directory.  
     -a,  --assembly=[VALUE]     The full path to the assembly containing your tests.  
     -r,  --results=[VALUE]      This is the full path to an .xml file that will contain the results. 
@@ -21,38 +33,66 @@ A console application which allows running RTF without a user interface. If you'
          --dry                  Conduct a dry run. (OPTIONAL)  
     -x,  --clean                Cleanup journal files after test completion. (OPTIONAL)   
          --continuous           Run all selected tests in one Revit session. (OPTIONAL)  
+         --groupByModel         Run tests with same model without reopening the model for faster execution, requires --continuous. (OPTIONAL)
          --time                 The time, in milliseconds, after which RTF will close the testing process automatically. (OPTIONAL)  
     -d,  --debug                Should RTF attempt to attach to a debugger?. (OPTIONAL)  
     -h,  --help                 Show this message and exit. (OPTIONAL)  
 ```
 
-##### RevitTestFrameworkGUI.exe   
-Provides a visual interface for you to choose tests from a treeview and to visualize the results of the tests as they are run. The same settings provided in the command line argument help above are available in the UI. The UI also allows you to save your testing session.
+As an example, the following command:
+```
+RevitTestFrameworkConsole.exe --dir C:\MyTestDir -a MyTest.dll -r MyTestResults.xml -revit:"C:\Program Files\Autodesk\Revit 2019\Revit.exe" --continuous
+```
+will execute all tests in `MyTest.dll` located in `C:\MyTestDir` and place all results in `MyTestResults.xml` (in the same folder). It will use Revit 2019 as specified and will run all tests without shutting down Revit.
 
-The input fields to set the test assembly, the working directory, and the results file, as well as the tree view where available tests are displayed, support dragging and dropping of files and folders.
+The results of the run as well as any `Console.WriteLine` or `Console.Error.WriteLine` from test will be shown in the command window and written ot the results xml file, e.g.:
 
-![Image](https://user-images.githubusercontent.com/3942418/28271251-0822ae5c-6ad6-11e7-8028-4f2f5c03823e.png) 
+```
+Reading assembly: D:\MyTestDir\MyTest.dll
+Loaded test: Name: UnitTest1, Model Path: D:\MyTestDir\MyModel1.rvt (D:\MyTestDir\MyModel1.rvt)
+Loaded test: Name: UnitTest2, Model Path: D:\MyTestDir\MyModel2.rvt (D:\MyTestDir\MyModel2.rvt)
+Running D:\Projects\UpCodes\upcodes_revit\Tests\2019\bin\Debug\RTF_Batch_Test.txt
+Running UnitTest1 in BasicTests
+Success: UnitTest1 in BasicTests
+
+Running UnitTest2 in BasicTests
+Success: UnitTest2 in BasicTests
+
+Waiting for Revit to terminate
+WARNING: One or more journal files could not be deleted.
+```
 
-`File` - For saving and loading your local RTF config.
+The warning about some journal files not being deleted can be safely ignored.
 
-`Test Assembly` - Path to assembly containing the tests to run. E.g. RevitNodesTests.dll, RevitSystemTests.dll, RevitServicesTests.dll.
+### RevitTestFrameworkGUI.exe   
+Provides a visual interface for you to choose tests from a treeview and to visualize the results of the tests as they are run. The same settings provided in the command line argument help above are available in the UI. The UI also allows you to save your testing session.
 
-`Results File Path` - Path to the xml file containing the tests results. If it's an existing file, RTF will replace it.
+The input fields to set the test assembly, the working directory, and the results file, as well as the tree view where available tests are displayed, support dragging and dropping of files and folders.
 
-`Working Directory` - Path to where the testing Revit files are.
+RevitTestFrameworkGUI.exe can also take an argument for the test assembly path and will automatically fill all values on start so you can just hit `Run`.
 
-`Additional Resolution Directories` - Path to find Dynamo Core location when it is not set in the `Dynamo.Config` in DynamoRevit bin folder.
+![RTF](images/RTF_UI.PNG) 
 
+`File` - For saving and loading your local RTF config.  
+`Test Assembly` - Path to assembly containing the tests to run. E.g. RevitNodesTests.dll, RevitSystemTests.dll, RevitServicesTests.dll.  
+`Results File Path` - Path to the xml file containing the tests results. If it's an existing file, RTF will replace it.  
+`Working Directory` - Path to where the testing Revit files are.  
+`Additional Resolution Directories` - Path to find Dynamo Core location when it is not set in the `Dynamo.Config` in DynamoRevit bin folder.  
 `Debug` - Check this if you decide to launch a debug session.
+`Timeout` - in milliseconds, the maximum time for a test to run. If a test doesn't finish in this time, Revit will be forcefully terminated and new test session without the offending test will start (the offending test will be marked as `timedout`).  
+`Continuous` - same as above, run tests without restarting Revit for each test.  
+`GroupByModel` - same as above, run tests with the same model without reopening the model.  
 
 ## Results  
 
-The output file from a test run is an nunit-formatted results file compatible with many CI systems.
+The output file from a test run is an nunit-formatted results file compatible with many CI systems.  
+For documentation how to use RTF inside a CI system, see [this guide](docs/using_with_ci.md)
 
 ## Revit Versions
 
 This repo maintains branches to track the two most recently released versions of Revit and one un-released version of Revit. When new versions of Revit are released, branches tracking the oldest version of Revit supported will no longer be maintained. For example, when Revit 2016 is released, the Revit 2014 branch of RTF will no longer be maintained.  
-When testing, you should run the version of RTF corresponding to the version of Revit you are running. This will ensure that tests you have created, based on one Revit API, will correspond to the version of the API running on Revit.
+When testing, you should run the version of RTF corresponding to the version of Revit you are running. This will ensure that tests you have created, based on one Revit API, will correspond to the version of the API running on Revit.  
+However, the current version of RTF works and can be used to test within Revit 2017, 2018, and 2019.
 
 ## License
 

appveyor.yml.deleteme

@@ -0,0 +1,36 @@
+version: 1.19.{build}
+pull_requests:
+  do_not_increment_build_number: true
+branches:
+  only:
+  - mark/Revit2019
+skip_tags: true
+image: Visual Studio 2017
+configuration: Release
+assembly_info:
+  patch: true
+  file: '**\AssemblyInfo.*'
+  assembly_version: '{version}'
+  assembly_file_version: '{version}'
+  assembly_informational_version: '{version}'
+before_build:
+- cmd: nuget restore .\src\
+build:
+  project: .\src\RevitTestFramework.sln
+  verbosity: minimal
+after_build:
+- cmd: nuget pack src\RevitTestFramework.nuspec -Version %APPVEYOR_BUILD_VERSION%
+test:
+  assemblies:
+    only:
+    - .\bin\AnyCPU\Release\RunnerTests.dll
+artifacts:
+- path: RevitTestFramework*.nupkg
+  name: nupkg
+deploy:
+- provider: NuGet
+  api_key:
+    secure: cN7zY+VTH8QH103SZn3xen5q3ZNqDvB4a5JKcYTGzBfnFRzCn/kldBPip6ENyCFF
+  artifact: nupkg
+  on:
+    branch: mark/Revit2019

docs/using_with_ci.md

@@ -0,0 +1,128 @@
+# Configuring CI to use RTF
+
+This doc describes how to configure a CI system and use RevitTestFramework to run Revit tests on the CI machine.  
+This document will describe setup for Microsoft Visual Studio and AWS EC2, however, the general instructions will work on other systems, e.g. Jenkins, TeamCity, etc.
+
+1. **Setup Visual Studio Team Services**  
+Visual Studio Team Services serves as a master in our CI. You can also use Jenkins for this purpose as well.  
+You will need to host your own machine (steps below) for actually executing the build/test steps as that machine will have to have Revit installed (default VSTS machines, naturally, do not have it installed)
+    1. Create an account if you don't have one yet
+    1. Set up an agent pool (we have 2 pools one for general build/test and one for UI tests)
+    1. Setup a new build pipeline:
+        1. As an example, you can use one pipeline for build and one pipeline for UI tests
+        1. Connect to your code repository (git/bitbucket/etc), this will popup oauth
+        1. Specify an agent pool to run on 
+           An agent pool is a pool of machines that can service a paricular pipeline. Since, in this example, we have 2 pipelines, we will use the `Default` pool for one of them and create a new one, `UI Tests` for the Revit UI tests pipeline
+        1. Define the pipeline  
+           These are steps that the build agent will perform to execute the tests, as an example, here are the steps for the pipeline we use:
+           1. Get sources
+           2. VsTest Platform Installer
+           3. Build binaries (we use a PowerShell script to perform the build, but you can use the default MSBuild step)
+           4. Run Revit UI Tests  
+              We use a PowerShell script here as well, which essentially executes our RTF tests, e.g.:  
+              ```
+              RevitTestFrameworkConsole.exe --dir .\bin\Release -a MyTest.dll -r .\MyTestResults.xml -revit:"C:\Program Files\Autodesk\Revit 2019\Revit.exe" --continuous
+              ```
+           5. Publish Test Results  
+              This steps takes the XML file produced by RTF and publishes it so that VSTS understands which tests pass and which fail.  
+              Select *NUnit* for Test result format and your test result file, e.g. `.\MyTestResults.xml`
+        1. At this point you can also configure triggers for each build pipeline (e.g. a pull request, or a merge into a specific branch, etc)
+
+1. **Setup the build server**  
+Now we setup a build machine that will connect to VSTS above and execute the build pipeline  
+    1. Decide where to host the build machine (AWS/Azure/local/etc) and what Windows variant you want to use. We use AWS and run our CI on Windows Server 2016.
+    1. Spin up a new instance, I recommend at least 16GB of RAM and 4 processors with ~150gb of storage (`m5.xlarge` on AWS)  
+    *Security note:* Since you will need to remote into this machine, it needs to be open to the world. To mitigate security risks:
+        * make sure you have a strong password (or use automatically generated AWS password) 
+        * limit the IPs that are allowed to access this machine to the IP of your office (you can use whatsmyip.org)
+    1. Remote into your new instance (as `localhost\Administrator`)
+    1. Create a new **non-admin** account (e.g. `vsagent`) on the machine (make sure it's a strong password and you have it saved somewhere safe!)
+    1. Setup some convenience stuff
+        * Create a PowerShell shortcut on the desktop
+        * Create a shortcut on the desktop to launch CMD as `vsagent` user, e.g. a shortcut to:
+        `C:\tools\PsExec.exe -accepteula -w c:\agent -i -u vsagent cmd.exe`  
+        This will allow you to run programs as `vsagent` user while being logged in as `Administrator`
+    1. Download and install necessary software on the machine:  
+       *Note:* you will not be able to use IE on the remote machine (if it's Windows Server) because it is locked down. I don't recommend removing the lockdown or installing some other browsers - this machine should be secure!  
+       I recommend getting the link to each download on your personal computer, then pasting that link into a PowerShell on the remote, e.g.:
+       ```PowerShell
+       # Disable progress bar: makes downloads ~50x faster
+       $ProgressPreference = "SilentlyContinue"
+       
+       # Download the file
+       Invoke-WebRequest <PASTE URL> -OutFile <FileName>
+       ```
+        1. [Git](https://git-scm.com/download/win) (or other source control software needed to connect to your source code repository, use defaults)
+        1. [Visual Studio](https://www.visualstudio.com/downloads/), scroll down to [VS Build tools](https://www.visualstudio.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=15)  
+        Make sure to select the following components to be installed (in addition to defaults): 
+            * .NET for desktop
+            * Windows SDK (latest version)
+            * Click-once tools
+        1. [Wix](https://github.com/wixtoolset/wix3/releases/download/wix3111rtm/wix311.exe) or whatever other tool you need to build your installer/setup package.
+            * If you use Wix, you will need to enable .NET 3.5 on the server, you can do this in the *Turn on Optional Features* control panel.  
+            You will probably need to reboot after this.
+            * Install Wix
+            * You will need to add the following system-wide environment variables:
+            ```
+            WixCATargetsPath = "C:\Program Files (x86)\WiX Toolset v3.11\SDK\wix.ca.targets"
+            WixTargetsPath = "C:\Program Files (x86)\MSBuild\Microsoft\WiX\v3.x\wix.targets"
+            ```
+        1. [PsTools](https://docs.microsoft.com/en-us/sysinternals/downloads/pstools), simply extract the .zip into a folder, e.g. `c:\tools`
+        1. VsAgent click the *Download Agent* button on your VSTS account webpage. To install it, just unzip it to a new folder: `c:\agent`
+        1. Revit (we use 2019.1 for our CI)
+            * Once installed, you will need to run the Revit once to register it and dismiss any first run prompts (e.g. EULA, etc). This will require allowing Autodesk registration site to bypass IE security settings:
+            * Open an interactive session to the `vsagent` user with `psexec` (or the shortcut we created above):  
+            `C:\tools\PsExec.exe -i -u vsagent cmd.exe`
+            * In the CMD for `vsagent`, open Internet Explorer:  
+            `c:\Program Files\Internet Explorer\iexplore.exe`
+            * In IE, click on the gear/*Tools* icon, *Internet Options*, *Security*, *Trusted Sites*, *Sites*, and add `https://registeronce.autodesk.com` to the list.
+            * Close IE
+            * Now, from the same command window, run Revit, e.g.  
+            `C:\Program Files\Autodesk\Revit 2019\Revit.exe`
+            * Accept all prompts, login, whatever you need to do, then close Revit.
+            * For good measure, run Revit again and make sure no prompt show up - as they will cause issues with RTF
+    1. At this point we have a machine that can run our CI/Revit UI tests, let's make a snapshot (AMI) of it so we can easily spin up as many of these as we want. For AWS/AMI:
+        * (on the remote machine) Open `Ec2LaunchSettings`
+        * Keep all settings, and click *Shutdown with Sysprep*
+        * In the EC2 console, select the instance and chose *Image* -> *Create Image* from the *Actions* menu (give the image a descriptive name, e.g. `Revit Addin CI`)
+        * This will take a few minutes. Once done you can use this AMI to spin up as many worker agents as you want.
+
+1. Spin up a build server and connect it to Visual Studio Team Services
+    1. If using an AMI, use that AMI. Optionally, you can skip the AMI step and use the machine instance from above directly (though it's not recommended)  
+       Settings to consider:
+        * You can assign an AWS role to the machine such that it has credentials to access any private resources you may need (e.g. if you want to deploy data to an S3 bucket)
+        * You can create a security group which limits remote connection to your IP addresses and assign that security group to the machine
+    1. Once the machine is booted up, remote into it.
+       For credentials, use `localhost\Administrator` and the password you decrypt on AWS console (with the PEM file)
+    1. You may want to consider installing code signing certificate at this point if you are signing your Revit Addin (highly recommended)
+        1. Open CMD to `vsagent` (see steps above)
+        1. Open `certmgr` from CMD
+        1. Install the certificate into personal store
+    1. Finally, configure the Visual Studio agent, open a PowerShell window and:
+        ```PowerShell
+        cd c:\agent
+        .\config.cmd
+
+        # Follow the prompts
+        # Server url:          your VSTS url, e.g. https://[mybuild].visualstudio.com
+        # Auth type:           PAT
+        # PAT:                 (get it from https://[mybuild].visualstudio.com/_usersSettings/tokens)
+        # Pool:                use default for CI and UI Tests for Revit (or whatever pools you created above)
+        # Agent name:          e.g. REVIT-CI-1
+        # Work folder:         [default value]
+        # Run as service:      NO
+        # Configure autologon: YES
+        # Account to use:      vsagent [+ your password when prompted]
+        ```
+        Reboot.
+    1. Once rebooted you should see your new agent on Visual Studio Team Services webpage    
+    1. Queue up a new build and see if it works!
+
+1. Debugging when things go wrong...
+   In some cases, you just need to remote into the build machine to debug a broken build.  
+   Since the build runs as the `vsagent` account but `vsagent` account can remote into the machine, you will have to follow these steps:
+    1. Open AWS console, EC2 instances
+    2. Select the instance and click *Connect*
+    3. Use the PEM file to decrypt the password
+    4. Once connected (you will always connect with `localhost\Administrator` account), open *Task Manager* and click on *Users* then right click on `vsagent` and click connect.
+    5. Now you can run and debug on the live machine