Two years ago, I have started to work with the TFS and so with MsBuild and Continuous Integration. I have soon seen that I was missing some automation tasks and so I turned to SDC (.NET SDC Solution Build & Deployment Process & Tools) that was adding more or less 100 new tasks. More than enough !
Yes but in the end I was still missing some specific task and so I have started to create my owns. Here is how Jarod.Sdc.Extensions project is born ! I have decided to create a package around .NET SDC (migrated after a while to SBF - Solution Build Framework) and to create my tasks.
Here is the latest version of this library including a new task : MsTest. Redundant with the Microsoft's TestToolTask or SBF's MsTest task ?
No not really. First it allow you to work more low-level by playing directly with command lines MsTest launching, and then it allows you to "ignore Inconclusive tests".
The reason why ? In our environment, business analysts are describing or coding tests weeks or months in advance specifying for which version they will have to bi in application. To avoid loosing them or having difficulties to find them back, we don't want to use the Ignore attribute. So what ? We say they are inconclusive. (Note that we have "only" the Team Edition For Software Developpers, and so in the 2005 version, no test list manager is available).
What is the consequence, we have "ad vitam eternam" a red build because we always have some inconclusive tests. In our specific case, this is the only case we have inconclusive tests.
So this new task allow to consider that all the inconclusive tests will be passed. You will still see the good result in the test run (seeing passed and inconclusive tests) but your global build will turn green if it contains only passed tests or passed and inconclusive tests.
This task supports Visual Studio 2005 and Visual Studio 2008.
Want to have a try ? Find it here !
Note that this version has been released almost one month ago but I didn't got time to communicate sooner...
As you will be writing code, you will probably be writing tests. And you will probably come to writing internal methods to limit their visibility. And it will be necessary in lots of cases to test also these internal methods. How to do that ? You won't be able to access them from your test project.
It's not a big problem as Visual Studio 2005 (and 2008) will generate accessors for you using reflection. VS 2005 will generate a code file (that will have the big disadvantage that when you will change the name of a method, the file won't gets generated and your project will continue to compile. VS 2008 will generate an accessor DLL that will gets compiled (and so re-generated) at the same time you build your project.
So do we really need something else ? Definitely yes.
-
If you work in TDD, you will write your test when your method do not exist yet (and so neither your accessor) and writing without any mistake the name of the future accessor may be a bit tricky. Ok, it's not the main problem
-
You may - and you will - write code that cannot be tested via the generated accessors
-
VS 2005 will, in some cases, not generate a method accessor
-
VS 2008 will generate accessors for all the methods, but some won't work (at least in the Beta 2 version, and I have read that the accessor support will be increased in the final version, but I haven't been able to test it yet)
What happens so ? Let's imagine you write code like :
internal static IEnumerable<T> Distinct<T>(IEnumerable<T> list)
{
List<T> result = new List<T>();
foreach ( T item in list )
if ( !result.Contains(item) )
result.Add(item);
return result;
}
Which solution so we have so ? Either write your own accessor (we can call such a method using reflection) or you use another solution called "InternalsVisibleToAttribute". This attribute will make all your internal types visible to one or more specific assemblies and you will be able to call them as if they were public.
Note that when you ask Visual Studio to generate unit tests, it will propose to "honor the InternalsVisibleToAttribute", meaning that by default, it won't generate an accessor, but will add the tag to the assembly file.
To have more information about this attribute, refer to the online MSDN : http://msdn2.microsoft.com/en-us/library/system.runtime.compilerservices.internalsvisibletoattribute.aspx
Note also that if you are testing a strong named assembly you will need to
-
Add a strong name to your test assembly
-
Update the InternalsVisibleToAttribute (from your code project) to add the public key of the test project
How can we extract a public key from a DLL ?
Simply by using the "SN" tool. It is available in the .NET framework SDK, ie by default in the folder "%ProgramFiles%\Microsoft Visual Studio 8\SDK\v2.0\Bin".
You can use it as follow :
C:\>sn -Tp c:\WINDOWS\assembly\GAC_MSIL\Accessibility\2.0.0.0__b03f5f7f11d50a3a\Accessibility.dll
Microsoft (R) .NET Framework Strong Name Utility Version 2.0.50727.42
Copyright (c) Microsoft Corporation. All rights reserved.
Public key is
002400000480000094000000060200000024000052534131000400000100010007d1fa57c4aed9
f0a32e84aa0faefd0de9e8fd6aec8f87fb03766c834c99921eb23be79ad9d5dcc1dd9ad2361321
02900b723cf980957fc4e177108fc607774f29e8320e92ea05ece4e821c0a5efe8f1645c4c0c93
c1ab99285d622caa652c1dfad63d745d6f2de5f17e5eaf0fc4963d261c8a12436518206dc09334
4d5ad293
Public key token is b03f5f7f11d50a3a
Note that you will need to provide the full public key (and not the public key token) in the attribute to make it work correctly. So for example:
[assembly: InternalsVisibleTo("Jarod.Sdc.Extensions.Task.Test", "0024000004...518206dc093344d5ad293")]
if I was using the public key of the .NET framework.
Tonight I was working to see how it was possible to wrap the MsTest execution to avoid returning an error code when a test is inconclusive.
In fact the "problem" comes from the DLL "Microsoft.VisualStudio.QualityTools.Common" from the path "c:\program files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies".
This DLL defines an enum TestOutcome and a method TestOutcomeHelper.IsFail
How to overcome this? The simplest way is probably to detail more your MsTest command line to see this outcome result (which is stored in the .TRX file) to be able to analyse it.
To do that, here is typically what you could use :
mstest /testcontainer:myTestLibrary.dll /detail:outcometext
What is this detail switch used for ? Simply to extract automatically some values from the TRX file. And I have found this link tonight on the MSDN that give a first list of the possible properties : http://msdn2.microsoft.com/en-GB/library/ms182489(VS.80).aspx#detail
 |
|
I have been writing a very small introduction to the CITCON in this post
The conference is now finished and so it's time to speak a bit about it !
This edition was full of surprises for me. First because it's the first time I was going to CITCON (pronounce it "KIT-KON") and the first time I was participating in Open Space discussion. The principle ? Simple. No conference, nothing really planned in advance.
On friday evening, after every one (between 60 and 100 persons) has presented themselves in a few words, some people started to explain the subjects they would like to discuss about. All could propose a subject and all could vote for their favorite subject in order to plan after how the different talks will occur on the next day and in which room. Interesting concept. In the end CITCON is really an excahnge of point of views, good and bad experiences, between people that all have the same goals : learning and sharing.
|
 |
Second surprise : I won a book. Nice. I never win to lotteries I participate into, but I win in lotteries I don't participate in ! 
It will probably be an interesting reading.
You want to see more of this book ? You can go to see it on Amazon
|
|
But let's come back to the discussions I have participated into.
-
At 10h00 : Discussions about the "Build pipeline" with maybe 30 participants, or how, why and which difficulties shall we encounter, to provide an automated build from development to automated test platform, QA platform and production platform. We have been discussing how recent is the technology, and so how "immature" or "not yet ready" are people from Operations. I specially liked the discussion and point of view of Xavier Quesada Allue about database update.
-
At 11h15, Jeffrey Fredrick (one of the organizer of the CITCON) has presented Crap4J, a new utility used for Java to determine where "crap is". By "crap", you should hear "risk", as this acronym means "Change Risk Analysis Prediction". How it works ? Putting in common the cyclomatic complexity and the code coverage to see the most potential risk in code.
-
At 14h00, Alexander Snaps had organized a small talk about "Code Reviews and Delayed Commits". This discussion involved a few people only among Dario Garcia Coder, Andrew Binstock, Patrick Smith and myself. It was for me, without any comparison, the most interesting discussion (maybe also because it was with much less people) with some opposite point of views. Shall we - or not - make code reviews before commits? If yes, on wich criteria shall we do them? Is it possible to ask the developpers to commit anyway (in a separate repository ?) and ask a code review later on ?
-
At 15h15, Douglas Squirrel wanted to share with us its idea of creating a new software : "Karma for Continuous Integration". Its goal ? Attributing (or removing) "Karma points" to developpers based on the commit he has just done (has he incresed te code coverage, decrease the complexity, ...). He wanted during this session to explain its idea and see what would be the people expectations about such an idea
-
And finally at 16h30, Bernard Vander Beken and Alexander Snaps wanted to speak about the Web UI Testing problematic. In the end we have spoken about some tools, possible limitations, how to test Ajax based application, should we go until "mocking up" the server itself ? Interesting talk wth the expertise of the google guys showing of what they have done.
What can I say more ? Well I just hope to continue the discussions with some of you guys and to be present in the next year session !
I will probably update soon this post to add some link to other interesting people I have met there !
|
| |
|
CITCON (Continuous Integration & Testing Conference) comes to Brussels in October 19th & 20th (friday evening and saturday).
Interested in 1-day of great conference about CI and testing ? Join me there !
Be careful, it's limited to the first 100 subscribee so hurry up.
Need more information about CITCON ? Visit their website dedicated to Brussels !
|