I have recently been looking at Docker, and how I can use TeamCity to run .NET Core unit tests in Docker containers as part of my build pipe-line. I add this as the final line in my Dockerfile to be able to run tests:
ENTRYPOINT ["dotnet", "test", "--verbosity=normal"]
These Dockerfiles are then referenced in compose files which TeamCity builds and runs using docker-compose in the command line.
I have this working successfully now. Next challenge is to break the build if unit/integration test coverage is less than 90% - or some other value - no arguments about this please!
I'm successfully using the coverlet.msbuild NuGet dependency to measure code-coverage as part of my build. This works fine in TeamCity too, and I see the output in my TeamCity build.
I got this working by adding coverlet.msbuild to each of my test projects, and changing the Dockerfile entry-point to:
ENTRYPOINT ["dotnet", "test", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=90", "/p:ThresholdType=line"]
The TeamCity build output shows the ASCII tables with the results in, but as of yet I've not been able to find a good way to break the build if the code-coverage isn't high enough. Left to its own devices, TeamCity doesn't mark builds as failing if the code-coverage is too low, which if fair enough as it's not psychic!
I naively thought I could create a failure condition in TeamCity which would detect for the presence of the following text:
'[Assemnbly]' has a line coverage '9.8%' below specified threshold '95%'
...using a regular expression like this:
has a line coverage '((\d+(\.\d*)?)|(\.\d+))%' below specified threshold '((\d+(\.\d*)?)|(\.\d+))%'
However, when DLLs being tested reference other DLLs that are tested separately, it gets tricky because coverlet.msbuild reports coverage metrics for all "touched" DLLs. For instance, I have a test project called Steve.Core.Files.Tests which tests Steve.Core.Files. However, Steve.Core.Files in turn references Steve.Core.Extensions. I test Steve.Core.Extensions separately in its own test DLL so I don't care about the results for that DLL when testing files. The output in TeamCity looks like this:
+-----------------------+--------+--------+--------+
| Module | Line | Branch | Method |
+-----------------------+--------+--------+--------+
| Steve.Core.Extensions | 23.5% | 40% | 40% |
+-----------------------+--------+--------+--------+
| Steve.Core.Files | 100% | 100% | 100% |
+-----------------------+--------+--------+--------+
...so it fails based on the 23.5% bit, even though the DLL in question is 100%. This actually makes it very difficult to check the using a Regex failure condition.
To complicate things further I'm running all tests in all assemblies using a single dynamic Dockerfile, for two reasons:
I don't want to have to change the Dockerfile and docker-compose file (and TeamCity) each time I add more projects and tests.
There are many dependencies between the DLLs so it makes sense to build them once and test them all together.
This means I'm loath to split the tests up so that each has its own Dockerfile - I know that this would allow me to use the Exclude/Include flags to get the desired behaviour.
Does anyone have any other ideas how I can solve this please?
I'm hoping I can add a file at the level of each test project to tell it which DLLs to do coverage for - that would be the best solution. Failing that, as I use a strict naming convention between projects and test projects, can I add a switch to my dotnet test command to test only the assembly that has the same name as the test assembly minus the .Tests bit on the end?
Thanks in advance; help appreciated!
Cheers,
Steve.
Update 7th September 2018:
So, my Dockerfiles are now specific to each unit test project. They look like this and exist in next to the test project files:
FROM microsoft/dotnet:2-sdk
# Set the working directory:
WORKDIR /src
# Copy the solution file and the NuGet.config across to the src directory:
COPY *.sln NuGet.config ./
# Copy the main source project files to the root level:
COPY */*.csproj ./
# Make directories for each project file and move the project file to the correct place:
RUN for file in $(ls *.csproj); do mkdir -p ${file%.*}/ && mv $file ${file%.*}/; done
# Restore dependencies:
RUN dotnet restore
# Copy all files so that we have all everything ready to compile:
COPY . .
# Set the flag to tell TeamCity that these are unit tests:
ENV TEAMCITY_PROJECT_NAME = ${TEAMCITY_PROJECT_NAME}
# Run the tests:
ENTRYPOINT ["dotnet", "test", "Steve.Core.Configuration.Tests/Steve.Core.Configuration.Tests.csproj", "--verbosity=normal", "/p:CollectCoverage=true", "/p:Threshold=95", "/p:ThresholdType=line", "/p:Exclude=\"[Steve.Core.Testing]*\""]
Note the exclude switch which is supposed to stop the coverage results for the Steve.Core.Testing DLL being included in the results for Steve.Core.Configuration, which is the main dependency of the tests, and the project being unit tested.
My compose file looks like this and exists next to the solution file:
version: '3.6'
services:
# Dependencies:
steve.core.ldap.tests.ldap:
image: osixia/openldap
container_name: steve.core.ldap.tests.ldap
environment:
LDAP_ORGANISATION: Steve
LDAP_DOMAIN: steve.com
LDAP_ADMIN_PASSWORD: Password1
steve.core.data.mysql.tests.database:
image: mysql
container_name: steve.core.data.mysql.tests.database
command: mysqld --default-authentication-plugin=mysql_native_password
environment:
- MYSQL_ROOT_PASSWORD=Password1
- MYSQL_DATABASE=testdb
steve.core.data.sqlserver.tests.database:
image: microsoft/mssql-server-linux
container_name: steve.core.data.sqlserver.tests.database
environment:
- MSSQL_SA_PASSWORD=Password1
- ACCEPT_EULA=Y
- MSSQL_PID=Developer
steve.core.email.tests.smtp:
image: mailhog/mailhog
container_name: steve.core.email.tests.smtp
# Steve.Core.Configuration:
steve.core.configuration.tests:
image: steve.core.configuration.tests:tests
build:
context: .
dockerfile: Steve.Core.Configuration.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Data.MySql:
steve.core.data.mysql.tests:
image: steve.core.data.mysql.tests:tests
build:
context: .
dockerfile: Steve.Core.Data.MySql.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Data.SqlServer:
steve.core.data.sqlserver.tests:
image: steve.core.data.sqlserver.tests:tests
build:
context: .
dockerfile: Steve.Core.Data.SqlServer.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Data:
steve.core.data.tests:
image: steve.core.data.tests:tests
build:
context: .
dockerfile: Steve.Core.Data.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Email:
steve.core.email.tests:
image: steve.core.email.tests:tests
build:
context: .
dockerfile: Steve.Core.Email.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Encryption:
steve.core.encryption.tests:
image: steve.core.encryption.tests:tests
build:
context: .
dockerfile: Steve.Core.Encryption.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Execution:
steve.core.execution.tests:
image: steve.core.execution.tests:tests
build:
context: .
dockerfile: Steve.Core.Execution.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Extensions:
steve.core.extensions.tests:
image: steve.core.extensions.tests:tests
build:
context: .
dockerfile: Steve.Core.Extensions.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Files:
steve.core.files.tests:
image: steve.core.files.tests:tests
build:
context: .
dockerfile: Steve.Core.Files.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Ldap:
steve.core.ldap.tests:
image: steve.core.ldap.tests:tests
build:
context: .
dockerfile: Steve.Core.Ldap.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Maths:
steve.core.maths.tests:
image: steve.core.maths.tests:tests
build:
context: .
dockerfile: Steve.Core.Maths.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
# Steve.Core.Time:
steve.core.time.tests:
image: steve.core.time.tests:tests
build:
context: .
dockerfile: Steve.Core.Time.Tests/Dockerfile
environment:
- TEAMCITY_PROJECT_NAME
When it runs in TeamCity, it reports only 7 tests from two projects (for some strange reason) even though there are 236 tests in 12 projects.
I'd be happy to email the output from the TeamCity build if it will help.
Does anyone know how I can get my tests all running again please?
Thanks,
Steve.
