Tag Archives: functional testing

Standard

Posted by

Posted on

December 29, 2014

Posted under

Load & Performance testing, Software testing, Test automation

Comments

9 Comments

JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication

What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

Standard

Posted by

Posted on

April 4, 2013

Posted under

Load & Performance testing, Software testing

Comments

Leave a comment

Requirements for Load & Performance testing

In my current assignment I am now tasked with writing a performance test plan for testing the performance of an ERP system. In this case, with performance, I mean the actual user experience which needs to be measured. So how long does it take for a screen to be fully rendered within a desktop application for a certain user-type with specific authentication and authorization.

Test data

Considering the organization I am still working in at the moment, the test data will also be an interesting challenge  the requirements we have for data are fairly simple, at first glance at least:

  • 80.000 active users within the ERP
  • 150.000 inactive users within the ERP

However when we start looking at the specifics of how this data is to be built, it becomes a bit more complicated. These users are divided over two organisations, where one organisation is a relatively simple pyramid structure, however the other organisation consists of a huge set (300+) of separate, smaller organisations, which have very flat organizational structures.
Generating this data is going to be fun! Especially since these users will need to be actual active users within the system, with an employment history, since the execution of the performance tests needs historical data to be available, not to forget that the users need to login during the performance tests and actively generate some load on the ERP system.

Performance requirements

Next up is the actual requirements we will be testing for. Some of the questions which popped into my head were:

  • How do you come up with proper L&P requirements?
  • What are bad requirements?
  • How do you get your requirements SMART?
  • How do you then measure these requirements?

So, we had several sessions with the end-users of the system to get to a basic understanding of what they required the application to follow and some really nice requirements came up, for example:

The application should finish a batch-job for at least 500 analyses within a time frame of 8 hours

Considering what SMART stands for this requirement leaves some gaping holes. Sure, it is Timely. The others however are not quite met yet.

This is merely one of many requirements we had to go through to make them SMART. The challenge in the requirements was mainly getting clear to the requirements owners what the difference is, from a testing and specifically performance testing point of view between the original requirements and the actual SMART version of that same requirement.

The example as stated above ended up as the following requirement:

The batch-job for executing predefined data analyses has to finish processing 500 separate analyses within a nightly run of 8 hours, after which the analyses  results are successfully uploaded to the end-user dashboards.

Getting requirements clear for batch-jobs however, is not the most difficult part. The main issue was getting the requirements clear for the user interactions and separating the desktop client interactions from the web-interface.

How do you explain in layman’s terms why a desktop application will by design respond faster than an average web-application and thus that you need different specifications for the two? How do you make clear, again in layman’s terms, that setting up the performance tests for a web-application will not make the scripts reusable for the desktop application, despite them having identical functionality, or even look and feel?

Those are some of the questions I have been struggling with the last few days while writing the performance testplan and with that defining and refining the requirements. (Why is it by the way that I, as the performance tester and defining the requirements I need to test for??)

Standard

Posted by

Posted on

October 27, 2011

Posted under

Software testing

Comments

20 Comments

Where are the technically inclined testers?

–Edit–

This would have been better had it been called something towards “Where are the technically curious testers” as Anna Baik pointed out. The wording is not perfect.

//Edit//

Over the years I have worked with a whole range of testers, both functional and technical, who have shown different levels of comfort with the technology under test. Some were good, some were great and others I simply never want to work with ever again. The latter ones, to me, are a problem we are currently facing in software testing. There seem to be too many software testers with a lack of understanding how software is actually built, how the internals of a program work.

I have never understood how one can be working in technology, or to be more specific in information technology and not have at least a base grip on how things you are supposed to test work? Even when you are purely focussed on functional testing, I really do not understand how you can not grasp the basics this entire industry is based on.

How come that even now, when it is becoming more and more clear that software testing is an important specialisation within the IT industry we still see a lot of testers with no technical knowledge what so ever in this industry? I realize that historically there was the idea that “anyone can be a software tester”.  I am just not convinced that statement still holds true.

Take for example a building inspector, that is also a tester of sorts, no? He verifies whether all the rules have been followed of proper architecture, the design has been implemented as promised etc. Imagine you are having your own house constructed and this building inspector comes by. You put your hopes on this person to validate that your house is what you want it to be: warm, safe and as you requested.

This building inspector, upon arriving, grabs his excel sheet with the requirements for the house and checks the ones he can verify, without needing any knowledge of architecture or constructing a house for that matter. He then gives you this checklist and tells you he has functionally covered all things of your house.

Considering that you live in an area with lots of snowfall in winter you ask this person whether the roof can manage with a load of snow on it in winter and the heating will not collapse under the strain of having to heat the entire house in a snow-storm.

He says: I don’t know, I am not an architect nor a construction worker, I am merely the building inspector. I just go through this list of things I need to check, why would I know anything about building a roof for a house or what the heating can take? You should probably ask the construction workers what they think.

Would this give you faith in his verdict that the house is indeed what you wanted?

I believe not, yet this is a practice that i see happening fairly often in software testing. So where does it still go wrong in software testing? How come there is still this tendency to believe there is no need for understanding software when you test it?

How often do you see the same bug coming by when testing an application, once you see a bug more than once, in a very similar form, you should be able to come up with the idea that there might be an underlying issue going on. Instead what I  often see is that for every occurrence a new bug is created in the bugtracker. To understand that there quite likely is one underlying problem, the tester doesn’t need to know how to program, you should however have a basic idea of how a program is (or should be) built up and if you are not certain whether it indeed might be one and the same issue in the code, how about talking to the developer?

When exchanging ideas and thoughts with other testers on twitter and forums etc I quite often see an amazing lack of knowledge in this area. To make things even worse, a fair share of testers seems to have a degree (bachelor or master) in computer science yet have no clue what, for example, a regular expression is. Is it just me, or is this indeed a worrying thing? When reading through the Computer Science curriculum of an average university here in the Netherlands, I do see all kinds of interesting subjects and descriptions that would lead me to believe basic programming is part of what you get taught, however when talking to the graduates that end up in software testing I see nothing of that knowledge.

Where are the technically strong testers? The ones that can have a discussion with developers about how the structure of a program was setup, who can tell a developer that the SQL query he wrote is extremely inefficient? I know I see some of them online, these are the testers I enjoy following on twitter and on blogs, but there must be more than these happy and noisy few. Where are they hiding? In my experience there are not enough of them, at least not in the Netherlands.

— Edit–

There is a nice article dealing with similar questions and frustrations on testnieuws.nl: http://www.testnieuws.nl/2011/06/06/tester-praat-ook-eens-met-een-ontwikkelaar/ (sorry, it is in Dutch, if you do want to attempt have a look at the google-translated version)

 

–Edit 03/11/2011 —

Really nice to read Elisabeth Hendrikson’s article on a similar subject, but from a different point of view: http://testobsessed.com/blog/2010/10/20/testers-code/

I am sad to see though that there are people commenting on her article and are calling QTP and especially Selenium basically record/playback tools. If you have ever used either you know that QTP is a hell of a lot more than just record playback and Selenium is clearly NOT a record playback tool (unless you mean the Selenium IDE rather than the entire toolset of Selenium.

 

 

 

Standard

Posted by

Posted on

September 6, 2011

Posted under

Test automation

Comments

Leave a comment

FitNesse – Test automation in a wiki

the assignment
When I started working at my current assignment I got told that the tools for automation had already been chosen, one of the tools being FitNesse. The way the tools were chosen is not my preferred way of picking a tool, the fact that the assignment was to “implement test automation across the organization through the use of the chosen tools” made me slightly worried whether or not FitNesse and the rest would indeed turn out to be the right choice.

Prior to this assignment I had heard a lot about FitNesse but had never had any hands-on experience with it, nor did I know anyone with hands-on experience with it.
Having worked with FitNesse for a few months now i feel the time has come to share my thoughts on it, what do I like, what do I believe is up for improvement, how is it working for me for now etc.

learning curve
Getting started with FitNesse was not all too intuitive. Getting it started is easy enough, but once you have it running it is not clear where to start and where to go from the FrontPage. Since we were not planning to use the standard fixtures but instead were planning to create our own we started on the fixture side rather than with FitNesse itself directly. Having created a generic login functionality in the fixture translating actions back int FitNesse became a lot more intuitive.

possibilities
The base fixtures such as the DoFixture, WebFixture etc. are very powerful in itself, I however feel they somewhat miss the point of automating in clear text: the tests are not easy to read, logical to follow or intuitive to write. We chose to work with SLIM rather than with FIT since FIT gives too much flexibility in usage of (almost) psuedo-code. Examples as used in the acceptance test in FitNesse are not clear enough for our purpose at this client. The test team is, to say the least, not very technically inclined and examples such as below do not really help them very much:

This is still somewhat readable

!|Response Examiner.|
|type  |pattern|matches?|contents?|
|contents|Location: LinkingPage\?properties|true||

A while loop implemented in FitNesse however quickly turns into black-magic in the hands of the technically less inclined:

|While|i=0|i<5|i++|
|put book in the cart|i|
|confirm selection|i|

With our custom implementation we now have test cases that can be read by most people within the organization and will be quite well understood, for example the below scenario for transferring money from one account to another:

|Scenario|Transfer Money|amount|From Account|accountFrom|To Account|accountTo|With Description|desc|
|Start               |TransferMoney |
|Go To|
|Select Van Rekening |@accountFrom |From Dropdown|
|Select Naar Rekening|@accountTo|From Dropdown|
|Enter Bedrag        |@amount|In Textbox|
|Enter Omschrijving  |@desc|In Textbox|
|Click On Verwerken|
|Select Content Frame|
|Is Text             |Het effect van deze overboeking op uw vrije bestedingsruimte is|Present|
|Click On Verwerken|
|Start               |CurrentAccount|
|go to|
|check               |Get data from column|Bedrag|3|

flexibility
Having started with Selenium as the driver below FitNesse enabled us to quickly build out quite a lot of test cases for the web applications. Part of the beauty of FitNesse in my opinion is that it is driver agnostic. In other words, it doesn’t really care what the system under test is, be it a website, a JavaApplet, a database or desktop applications. We are currently starting to work on TIBCO interfaces and will soon have to move over to Delphi and C# desktop applications. With quite some traditional test-automation-frameworks this would force us to start working either with a different tool or at least in quite a different way. The great thing about FitNesse is that it is so flexible that we can not only test desktop applications, we can also test across several parts of the platform. For example start executing some functions on the web application, verify these actions directly in the database, start a management desktop application and approve the actions initiated from the web application, all within one test case. A test case that big would make the test fragile, but the great thing is, it is possible if you really would want to.

refactoring
Quite some of the tests currently in FitNesse have been built up based on a functional mapping we initially made of the system, rather than the flows through the application. This is not quite ideal when running the tests, let alone when trying to sort through them and building up a suite for a particular type of flow or functionality.
Refactoring in FitNesse is probably the part where I believe a lot of improvements can be made. The current functionality, based on regular expression search is fairly crude.
FitNesse being a wiki, does have a wonderful different perk when needing to execute some bigger refactoring or moving around of test cases. All tests are text files within directories in the filesystem of your PC. In other words, if the built-in refactor function is too crude, a tool like Notepad++ or TextPad can be of immense value. These can search through files across directory structures and do a big part of the refactoring work for you. If you need to move whole folder structures, again you can copy them around directly on the file system.

judgement
My feeling regarding FitNesse so far is that it is a great tool which thus far seems to be underestimated out there in the world of test automation. Even when working directly with the standard fixtures FitNesse makes for easy to use, simple and quick to implement test automation. The main challenge is the initial learning curve, getting started with it and making the right choice in whether to go with Fit or Slim are for the newcomer the main hurdles.