Allure report

Details
Category: Technologies other

Reports has one of the most important role in automation tests development. If the author of the tests code could often understand the results of tests without report, for other team members it could be difficult. But autotests are written to make life easier to all around. 

In this article we will tell you about one of the possible systems for automations tests results reporting - Allure.

 

 

Now there are many systems in the world of automation that allows you to build a more or less pretty and understandable reports. You could use standart TestNG reporting or add Thucydides to your project. And not long ago there appears a new player - Allure. All useful information could be found on the official website of the product, which looks like test report. You can find an example of how the test results will look like and good and clear documentation. Also we want to note,  that the product is opensource and code is available to anyone and can easily modified to fit your needs.

OK, let's finish with lyric and  go directly to the product and look at it's features. Of course, with examples.

What is Allure?

At the begining of our article we want to pay your attention on one fact. Allure is the system for building report for single tests run, not more. Many peoples think, that it integrated with various frameworks like Selenium and so on. This is not true. Allure is just a framework that allows you to create simple and pretty reports, and you can use it with many products, frameworks and even programming languages.

Let's start.

Before we start, we want to say, that all examples in this article based on 1.4.14 version of allure. In later versions you could find more cool features, but the main functionality, as we think,  will be the same as in 1.4.14 version. If not - we'll update information immediately.

At the begining, as always, let's see how to connect Allure to your project . In our example, we will work with a bunch of JUnit and maven.

First, let's create an empty test maven project(you can use instructions to understand, how to do this job)

In our case, we used the following command:

mvn archetype:generate -DgroupId=com.perfecttest.allure -DartifactId=allure-test -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

So, our project was created. Now let's go on and setup allure.

For this, you should do the next manipulations in your pom.xml file:

1)add following settings to 'properties' section:

<properties>
    ...
    <allure.version>1.4.14</allure.version>
    <aspectj.version>1.7.4</aspectj.version>
</properties>

allure.version is not required field, but this way of defining versions is very good practice, because you have all versions in one place.

aspectj.version is required. Generally, AspectJ plays an important role in Allure. With this lib allure reads values of different annotations, used in report content. We'll  talk about them in our article.

2)in section 'builid'->'plugins' add maven-surefire-plugin, configured in special way  :

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.14</version>
    <configuration>
        <argLine>
            -javaagent:${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar
        </argLine>
        <properties>
            <property>
                <name>listener</name>
                <value>ru.yandex.qatools.allure.junit.AllureRunListener</value>
            </property>
        </properties>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>org.aspectj</groupId>
            <artifactId>aspectjweaver</artifactId>
            <version>${aspectj.version}</version>
        </dependency>
    </dependencies>
</plugin>

Here we could see reference to aspectJ version. Also, we want to pay attention on the listener configuration. For JUnit allure works through RunListener mechanism - added some class whose methods are invoked on specific events. If you want to add some logic in allure work process - no one prevent you to create your own implementation of RunListener, extended from AllureRunListener.

3)add  reporting section:

<reporting>
    <excludeDefaults>true</excludeDefaults>
    <plugins>
        <plugin>
            <groupId>ru.yandex.qatools.allure</groupId>
            <artifactId>allure-maven-plugin</artifactId>
            <version>${allure.version}</version>
        </plugin>
    </plugins>
</reporting>

4)add allure-junit-adaptor dependency. 

<dependency>
  <groupId>ru.yandex.qatools.allure</groupId>
  <artifactId>allure-junit-adaptor</artifactId>
  <version>${allure.version}</version>
</dependency>

This dependency contains all nedded classes for JUnit and allure communication.

That's all! Allure is now configured!

 

Tests.

Now, let's try to understand, what features are provided by allure. Of course, with examples of tests.

Let's create in test/java/com/perfecttest/allure folder class SimpleTest(don't forget about JUnit naming conventions)

Now, let's write very simple test that will check that 2 * 2 = 4. It's code will be :

@Test
public void firstSimpleTest() {
    assertTrue("Result not equals to 4", 2 * 2 == 4);
}

We have test, now we all interested, what we'll see in allure report for this test? To run this test, run the following command from cmd:

mvn clean test site

where

clean - clean your working directory(it will delete all "garbage" from previously build)

test - run all your tests from 'test' directory

site - generate report.

And what do we get?

Open the file target / site / allure-maven-plugin / index.html and you will see your first allure report!. It looks like this:

 

Ok. What you could see on this screen?

  1. Suites list. As suite allure takes test class. It's name takes as divided on the principle of camelCase class name. 
  2. Tests duration. Very useful indicator. When you have a lot of tests - you could understand, which tests are longer than the others and may need to be optimized.
  3. The number of tests in different statuses(failed,canceled, pending, total). In our case there is only one test and it was successful, so we don't see any information in statuses columns.  Also, here you could show or hide tests with specified statuses. To do this, press on the button with corresponding status.
  4. On the left you could see some interesting tabs with different graphs and errors lists.

Now let's describe what every status means:

  • Failed - this is failed tests.
  • Broken - tests, which are failed not on assert, but because of unexpected error or exception, occurred due to tests run.
  • Canceled - skipped tests(for example - when you call  assumeTrue(false) in JUnit)
  • Pending - ignored tests(marked with @Ignored in JUnit)

Let's click on the suite name(don't be afraid, nothing bad will happen). From right side of the screen comes the list of tests. Each name of the test based on camelCase mechanism.

Now we could see the following in browser window:

 

There we could see the test duration, and also the full path to the test class(it's near the suite name). Sometimes it could be useful - when some test fails and you want to see what happened in code - you could just select the name and find it with search in your favourite IDE.

Now let's go further and click the test name. Now we could see the steps of the test. 

In our case - there are no steps in test, so this section is empty. But it's only now. At the end of this article, there will be many really cool things!

After look at these 3 screens, it can be concluded that the structure of the project in allure is the following:

There are test suites(classes), they consist of test cases(methods in classes), they consist of steps(methods)

Once we figured out about global concepts allure, let's look deeper inside.

Main features in allure achieved by using annotations, placed at various locations of the code. Let's talk about them in more details.

 

@Title

We, as well as, perhaps, you want to make the report clear and beautiful as possible, as it could be.

And first what we are going to fix will be the names of suites and cases. Allure gives you an opportunity to do this in very simple way - all you need is just add @Title annotation to class or method. Let's do this in our test. Now our test code will be as follows:

@Title("This is our cool test suite")
public class SimpleTest {

    @Title("First cool check")
    @Test
    public void firstSimpleTest() {
        assertTrue("Result not equals to 4", 2 * 2 == 4);
    }
}

Now, after running our test we could see, that suite and test changed their names:

 

Cool, isn't it? Let's go on!

 

@Description

Since we want to make our report perfect - let's add some more descriptions that allows our colleagues to understand what to do in the suite or in the test. For this allure has @Description annotation. It can be added to the test class and or test method. Let's do it! Now our code is as follows:

@Title("This is our cool test suite")
@Description("In this cool suite we will test only cool features")
public class SimpleTest {

    @Title("First cool check")
    @Description("In this cool test we will check cool thing")
    @Test
    public void firstSimpleTest() {
        assertTrue("Result not equals to 4", 2 * 2 == 4);
    }
}

After running  the test , we could see several new blocks:

Now it becomes a little clearer, what we do in selected suite or case. 

@Severity

Sometimes, you want to add the severity to test to indicate, which is more or less important. On one of the screenshots above you could see the Severity near the test name. In allure there is an @Severity annotation, which may have one of these values: BLOCKER, CRITICAL, NORMAL, MINOR, TRIVIAL.

@Features and @Stories

Often it would be desirable to combine different suites (or tests) into something more global. For example, we write the tests for two different functionalities: receiving payments and user registration. As we now know, we could combine our test in suites, but now we want to have more than one suite for each functionality. For results grouping there are two annotations in allure - @Fatures and @Stories. If you add them to the test class or test name, then when report will be generated, you could open Behaviours tab on the left of the main report page and see some chart with tests, combine by features and stories. As in our case - it is possible to make feature "receiving payment", and stories : "payments from physical persons" and "payments from other cool guys"

@Step

Now we introduce you one of the main annotation in allure -  @Step. Remember, we talked about the steps? This annotation, added to any method, automatically makes it as step.

Let's add some methods and annotate them with Step:

@Title("This is our cool test suite")
@Description("In this cool suite we will test only cool features")
public class SimpleTest {

    @Severity(SeverityLevel.BLOCKER)
    @Title("First cool check")
    @Description("In this cool test we will check cool thing")
    @Test
    public void firstSimpleTest() {
        checkResult(getDivisionResult(4, 2), 2);
    }

    @Step("Check of calculation result")
    private void checkResult(int actualResult, int expectedResult) {
        assertTrue("Actual result("+ actualResult+") not equals to expected("+expectedResult+")", actualResult == expectedResult );
    }

    @Step("{0}")
    private void log(String value) {
        //empty method
    }

    @Step("Taking result of division {0} and {1}")
    private int getDivisionResult(int firstInt, int secondInt) {
        checkNotZero(secondInt);
        return firstInt / secondInt;
    }

    @Step("Check of divider")
    private void checkNotZero(int intValue) {
        log("Divider value: " + intValue);
        assertTrue("Divider is 0", intValue != 0);
    }
}

As we could see - there appears some new methods:

  • checkResult - method, checks the result of calculation
  • getDevisionResult - method, returns the result of division of two int's
  • checkNotZero - method that checks, that divider is not 0(we don't want to see BROKEN status of test in report)
  • log - this is util method that helps us with logging of some values as step in report.

Oh, but what does in means "{0}" and "{1}" inside annotation value? This is the way to use methods parameters values in step name. You can dynamically add or change the text of step "on the fly". While generating the report data, the {0} will be substituted with parameter.toString() value. Don't worry - developers don't forget about primitive types and null values. All will be OK with them.

We have done so many different methods in order to show that you could have many levels of nested steps.

Let's start our test and look at report:

 

See, all our {0} and {1} substituted by values of parameters! Cool, isn't it?

Now let's talk about another very useful thing in allure - attachments.

 

@Attachment

Allure gives you an opportunity to add attachments to your tests and steps. You could do this by using @Attachment annotation. Almost everything could be an attachment, but the most often use of this functionality is attachment of images(for example screenshots in Selenium), texts or html pages.

We'll describe all these three variants in examples and stops with more attention, because this type greatly expands your report potential.

@Attachment annotation has  2 parameters:

  • value - name of the attach. There you also use parameters and their indexes.
  • type - this is  mime type of attach.

Ok, let's start from the simplest - attach of the text file .

Let's write some helper method for attacment save:

@Attachment(value = "{0}", type = "text/plain")
public static String saveTextLog(String attachName, String message) {
    return message;
}

Now do some modifications with our test - remove some methods and add an attacment method call:

@Title("This is our cool test suite")
@Description("In this cool suite we will test only cool features")
public class SimpleTest {

    @Title("First cool check")
    @Description("In this cool test we will check cool thing")
    @Test
    public void firstSimpleTest() {
        checkResult(4 / 2, 2);
    }

    @Step("Check of calculation")
    private void checkResult(int actualResult, int expectedResult) {
        saveTextLog("Simple text attach", "This is sparta!!!");
        assertTrue("Actual result(" + actualResult + ") not equals to expected(" + expectedResult + ")", actualResult == expectedResult);
    }

    @Attachment(value = "{0}", type = "text/plain")
    public static String saveTextLog(String attachName, String message) {
        return message;
    }
}

 

As result we'll see:

 

As you could see - there are one more element in step. When click on it - the attachment file content opens. This is very useful when you want to log some requests and responses to some of your servers.

Second type - image attachment. Let's put the image to resources folder of our project and read it from there. 

Reading and attachment methods will looks like following:

@Attachment(value = "{0}", type = "image/png")
public static byte[] saveImageAttach(String attachName) {
    try {
        URL defaultImage = SimpleTest.class.getResource("/allure.png");
        File imageFile = new File(defaultImage.toURI());
        return toByteArray(imageFile);
    } catch (Exception e) {
        e.printStackTrace();
    }
    return new byte[0];
}

private static byte[] toByteArray(File file) throws IOException {
    return Files.readAllBytes(Paths.get(file.getPath()));
}

Code of our test now looks like this:

@Test
public void firstSimpleTest() {
    checkResult(4 / 2, 2);
}

@Step("Check calculation result")
private void checkResult(int actualResult, int expectedResult) {
    saveImageAttach("Image attach");
    assertTrue("Actual result(" + actualResult + ") not equals to expected(" + expectedResult + ")", actualResult == expectedResult);
}

 Now let's run test and look at report:

 

Yep! Image(in our case - that is allure logo) - is attached to step. 

And now - the sweetest part - html page attachment. Method looks like an image attachment, but with other type:

@Attachment(value = "{0}", type = "text/html")
public static byte[] saveHtmlAttach(String attachName) {
    try {
        URL defaultImage = SimpleTest.class.getResource("/test.html");
        File imageFile = new File(defaultImage.toURI());
        return toByteArray(imageFile);
    } catch (Exception e) {
        e.printStackTrace();
    }
    return new byte[0];
}

test.html, as image in previous example, lies in resources folder of the project. There could be any html. In our case - this is form with textbox and button, when clicked on it - displays the text from the textbox under the button. We used jquery and bootstrap in our html, that once again shows that it is possible to use many frameworks to add desired functionality to report through attach.

Also we would like to note, that attach of html could be done in another way, like text attachment:

@Attachment(value = "{0}", type = "text/html")
public static String saveHtmlAttach(String attachName, String html) {
    return html;
}

To use it in such way , we must transfer file content to string, but since we already have methods, that reads image from resource - let's use it to read html.

Now, our test have such code:

@Test
public void firstSimpleTest() {
    checkResult(4 / 2, 2);
}

@Step("Check of calculation")
private void checkResult(int actualResult, int expectedResult) {
    saveHtmlAttach("Html attach");
    assertTrue("Actual result(" + actualResult + ") not equals to expected(" + expectedResult + ")", actualResult == expectedResult);
}

And report:

 

Great, isn't it?

As we described earlier - html attachment opens up an a great possibilities. No one prevent you from generating of html while your java code running, or build some data and then  transfer it to html. With this features our report becomes realy powerful instrument for all, who uses it.

For example, you could do the next things:

  • Link your tests with bugs in bug tracking system. For this you should save the task number and test id(which could be transferred from java tests) into some storage, and then run/not run tests depend on task status. 
  • Configure test data. From such form like in our example you could setup some test data in storage, and then reads it from java while test execution.

and many many other cool things. 

 

Conclusion.

We hope, that  this article will helps you to understand this really great product. 

If you have some problems or ideas - contact as and we'll help you with great pleasure.

Also if you have some ideas about what to add with attachments - we could discuss it and develop together! Contact us.