Category Archives: Software testing

Posts about software testing in general

Standard

Posted by

Posted on

January 20, 2017

Posted under

Load & Performance testing, Software testing

Comments

Leave a comment

Jmeter Tips & Tricks – Tip 7

Tip 7 – adjusting the JVM HEAP size, why and how to

When running JMeter tests, it may happen that the test is not successful and JMeter crashes with an OutOfMemoryError. This can happen due to all kinds of things, most of the time however you are trying to generate too many threads with an insufficient amount of memory allocated to your Java Virtual Machine. More information on what the HEAP is, how the JVM deals with the HEAP can be found here.

The HEAP size is defined on startup of your java application, in our case when we start JMeter. In other words, adjusting the HEAP means we will need to adjust the startup script. The startup script can be found in your JMeter “bin” directory. In my case, on a linux environment, it can be found here:

/opt/apache-jmeter-3.1/bin

When opening the “jmeter” file in your favorite editor, scroll down until you hit a line stating the following (on Windows based systems open “jmeter.bat”):

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms512m -Xmx512m"

On Windows systems the line looks slightly different, in the jmeter.bat, due to the nature of BATCH files and DOS working differently from Linux systems

set HEAP=-Xms512m -Xmx512m

When you adjust this line to read something like this:

# This is the base heap size -- you may increase or decrease it to fit your
# system's memory availability:
HEAP="-Xms1024m -Xmx4096m"

the HEAP size will by default be larger, meaning you do not have to think about it anymore when starting JMeter. This does pose one possible challenge, it slows down the startup of JMeter somewhat (at least on most machines I have done it). This seems to be because the JVM will first allocate all the memory you defined in the Xms param, ensuring the JVM is at the appropriate size instantly.

So, we changed the values of Xms and Xmx, what do these two mean?

-Xms ==> minimum heap size. This is the value JAVA will attempt to claim in order to run JMeter. This does NOT imply that it immediately gets this fully allocated by the OS. The OS typically doesn't give an application all blocks it asks for until really needed.
-Xmx ==> maximum heap size. This is the value JAVA will use as a maximum size of the JVM. When JMeter passes this level, an OutOfMemoryError will occur once more.

An alternative, more flexible setup to change the heapsize is to pass the JVM arguments on the commandline when starting JMeter. This of course assumes you are comfortable in commandline and start JMeter from commandline.

[user@machine ]$ JVM_ARGS="-Xms1024m -Xmx2048m" jmeter

Be Aware!

Please always make sure you leave enough memory for your OS and other applications to run on your machine when allocating more HEAP space to a JVM. Not leaving enough memory for your OS to keep running will result in your machine simply being non-responsive and thus your test and testresults being rendered useless.

 

Standard

Posted by

Posted on

August 19, 2016

Posted under

Load & Performance testing, Software testing

Comments

1 Comment

Jmeter Tips & Tricks – Tip 6

Tip 6 – Effective use of Transaction Controllers

Whether you are trying to script a website, a webservice or something else, quite often you find that one functional action results in several requests to a server.

Take for example an average login-request seen from a functional point of view:

LoginScreenA user types in username & password and hits the Login button.
If the username/password combination was valid the user is than shown a dashboard.

This seems straight forward enough from a functional point of view. What actually happens however is a bit more than just this one request:

  • Username/password are sent to an authentication service
  • Authentication service sends an authenticated message back (assuming you are indeed authenticated by the application).
  • The authentication message contains a redirect away from the authentication service to the dashboard a user expects. This redirect contains information such as the AuthToken that was just recieved from the service.
  • Subsequent requests for the Dashboard-contents are sent.

In a simple setup, this may require only 1 request in Jmeter, just the login. Since Jmeter can take care of any redirecting itself.

Making Jmeter responsible for blindly following all redirects however, does not always give you the performance metrics you may want or need. When you need to know exactly which part of the login sequence is giving your users some form of grief, you may want to build these requests completely yourself in Jmeter. That will give you the possibility to see if there is high latency, big loads etc. on a per-request basis.

You may end up with something like this: jmeter-4requests

The 4 requests you see in this example are all separate requests used to log a user into the system and get the base information for the first screen of the application.

This sequence could have been captured in 1 request, Login, since the other requests are handled by HTTP/302 responses.

By building the requests yourself and clustering them together in a Logic Controller you now can see and measure how long the entire login request, step by step, takes. The Logic Controller I have used here is a Transaction Controller, this controller can add all child-transactions into one single response time for you by setting the appropriate flag. TransactionController
When set, “Generate parent sample” will ensure that your statistics have a single statistic for the complete set of requests, yet you can easily gain information on the separate requests while running a loadtest by unchecking this flag and looking into each step by itself as well.

This gives you the possiblity to find out which of the steps slows the entire transaction of logging in to your application (or whichever functional step you define in a Transaction Controller of course) .

Standard

Posted by

Posted on

June 26, 2015

Posted under

Software testing, Test automation

Comments

Leave a comment

Heb je die testautomationtool wel nodig?

3 tips om tot een goede toolselectie te komen

Als testconsultant kom ik vaak bij klanten over de vloer die een testautomatiseringstool willen aanschaffen. Ze stellen mij dan regelmatig de vraag “welke tool moeten we kopen of gebruiken?”.  Ik antwoord dan meestal met een tegenvraag: Waarom wil je een testautomatiseringstool hebben? De reactie die ik hierop krijg is redelijk voorspelbaar: grote ogen, gefronsde wenkbrauwen,  verbijstering op de gezichten dat een consultant zo’n onnozele vraag stelt en vervolgens de (steeds falende) poging om een antwoord te geven. GO BACKAntwoorden die ik krijg gaan vaak in de richting van:

  • “Ik heb er goede dingen over gehoord of gelezen”
  • “Een kennis van me werkt er ook mee”.

Of het nog gevaarlijker:

  • “Bij mijn vorige werkgever gebruikten we ook een testautomatiseringstool”.

Het mooiste antwoord wat ik ook nog wel eens krijg is:

  • “We hebben een demo gehad van een tool en dat zag er heel veelbelovend uit”.

The first principle is that you must not fool yourself — and you are the easiest person to fool.

– Richard Feynman

Deze antwoorden zijn duidelijke indicatoren dat het echte probleem niet duidelijk is en daarmee is ook het doel wat de klant hoopt te bereiken met de testautomatiseringtool niet duidelijk.

Wat is het probleem dat opgelost moet worden?

Waarom stel ik nou juist die vraag? Er zal toch wel een goede reden zijn waarom die klant vraagt om een testautomatiseringstool?

In veel gevallen is de reden van het zoeken naar testautomatiseringstools niet de échte reden, de diepere reden. Door te vragen waaróm men op zoek is naar testautomatiseringstools ga je op zoek naar welk probleem er is. Ik wil weten wat de aanleiding is voor het willen implementeren van een tool en vooral wat de klant hoopt ermee te bereiken.

Ik wil mijn gesprekspartner uitlokken (of uitnodigen) alles boven water te halen.

In veel gevallen is een testautomationtool in eerste instantie een lapmiddel. Het is symptoombestrijding omdat het eigenlijke probleem

  • te groot is
  • buiten jou invloedsfeer ligt
  • niet op korte termijn opgelost kan worden.

Symptoombestrijding geeft geen oplossing voor het echte probleem. Als bewuste symptoombestrijding het hoogst-haalbare is op dit moment, dan is het wellicht een goed idee om met een tool aan de slag te gaan. Vaak echter lijkt een testautomatiseringstool een oplossing te bieden voor iets wat niet de basis van het probleem is en dus wordt het echte probleem niet weggenomen, laat staan opgelost.

Er zijn een aantal dingen waar je naar moet kijken als je een testautomatiseringstool gaat selecteren.

 Wat moet je weten voor je een testautomatiseringstool selecteert?

  1. Definieer de basis van het probleem dat je poogt op te lossen met de testautomatiseringstool
  2. Stel jezelf de vraag “gaat een tool dit probleem daadwerkelijk oplossen?
  3. Hebben we de kennis, kunde en financiële middelen om een en ander ook door te zetten en het probleem écht op te lossen?

 Het is mogelijk om op tal van manieren te falen… terwijl je slechts op één manier kunt slagen.

– Aristoteles

Standard

Posted by

Posted on

December 29, 2014

Posted under

Load & Performance testing, Software testing, Test automation

Comments

9 Comments

JMeter and WebDriver – Why would you want to combine them?

3 Reasons to combine JMeter and WebDriver in performance testing a webapplication

What is JMeter good at?

The Apache Foundation states the following about JMeter:Apache JMeter

The Apache JMeter™ desktop application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

So in other words, it’s a tool you can use to generate functional load on an application or a platform. This also immediately describes what it is good at: generating load. Yes if implemented well you not only generate dumb-load but also hit the functional application layers with the tool. But the basic function of JMeter is aimed at generating load and measuring the (server) response times during this load.

What does JMeter NOT do?

Despite being capable of generating functional load, JMeter does not render pages nor is it very well equipped to execute embedded JavaScript (it is simply not equipped to do that actually), therefore JMeter will not tell you anything (out of the box) about the render times of pages. Especially not about render times when the server is heavily overloaded by your scripts!

What is WebDriver good at?

SeleniumHQ gives a wonderful description of what Selenium (or nowadays WebDriver) does:
Selenium Logo

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that.

In short, what Webdriver does is just about anything that happens within your browser. It does render pages, it does execute JavaScript, it retrieves the pages as if an actual human was clicking on a website. So for fully functional automated testing (or checking to stick with the more correct terminology) WebDriver is perfect.

What does WebDriver NOT do?

Well, it is not quite good at generating load. Since WebDriver basically requires a browser (yes, it is possible to run it headless of course) it is very difficult to generate multiple (virtual) users. That would require a bunch of browsers to start up, when talking about 10 users that may seem feasible, however when talking about generating real load (say several 1000’s concurrent users) a bunch of browsers becomes a lot more difficult to arrange.

Why combine them then?

The logical question then indeed is, why would you combine them? Below I have set out 3 clear reasons why combining JMeter and WebDriver scripts can be an excellent idea.

  • Impact of server-side load on render-times;
    When the load on a server increases, the response times of various parts of a web application may increase as well. These increased response times can have implications on the render time of the web application. For example: a web application heavy with Ajax requests is put under load, the server response times increase, this may result in all Ajax-requests becoming slower, therefore making the website extremely unfriendly to the end-user. When you just run a JMeter script, this will hardly be noticed, and if you do notice it, you cannot express the impact it will have on the user. You can merely speculate about it.
  • Impact of server-side load on functional behavior;
    Given that the server is experiencing increased load and therefore the business-logic of the application is working hard to handle all requests effectively, it can be safe to say the underlying database may also be stressed and therefore responding slower than expected. Slower response times of both application-logic and database requests can result in buggy behaviour of the application. For example incomplete data returned, or worse, a time-out on data or application-logic. How does the application deal with that? How are these errors reported to the end-user? Will the application still function normally within the browser when certain aspects of the application platform are malfunctioning? The best answer to this is by testing the functionality thoroughly while the application is under load. An easy way to test this repeatedly and consistently is by automating these functional tests, for example using (part of) the automated regression test while the servers are under increased load.
  • Advantage of screenshots of fully rendered pages and possible errors with the application under load;
    As a result of the two points mentioned above, it may be extremely useful for both developers/system engineers and your customer to see errors on the pages affected by the increased server load, such as stylesheets not loading or not loading properly, JavaScript not loading, images missing etc. Screenshots (or screencaptures in movie format) will help make clear to the customer what the problem is and more importantly how big the impact on the end-user will be.

I have listed 3 reasons why combining JMeter and WebDriver can be a good idea. I’d love to hear your suggestions of more reasons to want to combine the two.

In a follow-up post I will go into more detail on ways to achieve an effective combination of JMeter and WebDriver running along side each other, well timed and generating consistent logging and results.

Standard

Posted by

Posted on

June 20, 2013

Posted under

Software testing

Comments

12 Comments

Technical skills are a must for modern testers

In a previous post I already asked “where are the technically strong testers these days?“, in this post I would like to revisit that subject.

When looking at the current developments within IT the focus of testing is getting to be more and more on the business-logic and functionality rather than user-flows. More and more systems are delivered as API-based systems or service oriented architectures where the UI, or front-end, is only to a lesser extend important. UI’s can be put live into the world a lot easier, with béta testing, testing in production etc. Whereas the business-logic of an application, as well as the API’s are increasingly becoming more and more important, since that is where the actual value of most applications is.

traditional testingWhat does this mean for traditional testers? There already is a lot of talk about the changing role of testmanagers and coordinators in the future, what with Agile and things. But I believe the role of the ‘traditional functional tester’ will also lessen.

More and more testing will be pulled into the “technical” side of software, so into the services and API testing spheres, running functional tests through the API rather than through the user interface. This will result in testers, rather than being able to click their way through a user interface, will need to get used to working their ways through systems architectures and learn how to quickly and easily manipulate XML and other formats over a wide variety of protocols from one system to another. They will need to learn how to set up a full end to end chain and troubleshoot within this chain whenever an issue seems to appear.

Out of all the traditional functional testers I personally know, I am not sure what percentage will be able to make this transition, either due to lack of technical insight and understanding or to a simple lack of willingness to invest the time to learn some extra technical skills.

There are of course courses and trainings for testers to gain knowledge in the technical side of software testing, but whether that is enough.. I personally doubt it.
To be or become a good tester with technical knowledge and skills you will need to not only have an understanding of how systems are built up, but also a base understanding of programming. In order to gain these skills I believe more than formal training is required, you need to be willing to invest some extra time in it by home studying these concepts, practicing with them and keeping up with new technologies which might come your way sooner than you’d think.