Automate database load testing with Benchmark Factory®

Continuous Integration using Benchmark Factory’s REST API Part 1

[00:00:03.74] Hi, my name is Mathew Phan. I’m a software consultant at Quest Software, specializing in our database management tools, like Toad and Apex SQL. What you’re seeing is a popular continuous integration tool, Jenkins, currently in the middle of a build pipeline. And within my build pipeline, Jenkins is leveraging Benchmark Factory to automatically run a user load test against my Oracle database. This ensures that any new code I add to my application doesn’t affect the performance of my database.

[00:00:36.01] So in this first video of a two part video series, I’m going to talk a little bit about Benchmark Factory, what it does, and its REST API, as well as go through some examples. In the second video, I’ll show you how to leverage the REST API in scripts, and then put those scripts into CI Tools like Jenkins, or Bamboo, or Team City.

Understand your database and application performance while under stress

[00:01:01.37] Benchmark Factory is a database load testing tool. It can kick off thousands of virtual users connecting to your database and running a mix of transactions. While this load is being placed on your database by Benchmark Factory, it will display and record metrics like database response time and transactions per second. You can also use diagnostics or monitoring tools, like Spotlight or Foglight, and understand your database and application performance while under stress.

[00:01:31.53] Let’s imagine you’re a developer working on the landing page after a customer logs into your app. You’re tasked with implementing a new feature that displays the customer’s last three orders. Well, that’s now additional overhead to your database via the additional queries needed to get the user’s last three orders. And although you may be able to test just one user logging in fairly easily, it’s much more difficult to test thousands of users logging in concurrently in understanding the impact of this new feature that you’ve added.

[00:02:04.34] Compounding on this difficulty is that you may also want to run a workload of thousands of users already logged in, browsing the store, adding products to their shopping cart, checking on stock or the status of their orders. And you don’t want these already existing transactions to be impacted by the new feature that you’re adding. Starting in Benchmark Factory 7.6, a REST API was implemented. And with each iteration of Benchmark Factory since then, that API has been improved. Now, you can interface with Benchmark Factory from anywhere, from almost anything, giving it huge flexibility. Not only can you kick off a job and get back the test results, but you can also pretty much do anything from creating connections, building a whole BMF job, defining transactions, running it, of course, checking on the status and results of those jobs, and then deleting that job using the REST API.

You can access Benchmark Factory through its REST API to create jobs, run jobs, and get test results

[00:03:06.93] The Benchmark Factory REST API documentation is available online. It pretty much has everything you need to get started. To focus on our user forums at ToadWorld.com, they are also an excellent resource. The URL for the endpoint will be the hostname of where Benchmark Factory is installed or the IP address. The port number is going to be 30100. Then you’re going to go put a slash API.

[00:03:36.51] Each of the methods describes the endpoint as well as any parameters that are required. So let’s take a look at this first one. It’s a GET. And we’re going to get the slash version to get the version of Benchmark Factory that’s installed. I can do that from any browser, as well. So I can go to HTTP:// the local host, in my case. And the port number is going to be 30100 slash API slash version. That will return the version number of the Benchmark Factory I have installed in a JSON format.

Postman

[00:04:26.06] So let’s go over to Postman. Postman is an open source tool that enables me to quickly test an API. In this case, I’m going to be testing the Benchmark Factory API. It allows me to do things other than Get, like Post, Put, Delete, and so forth. So at the same command, HTTP://localhost :30100/API/version. I’ll go ahead and send that over to Benchmark Factory. And I get back the version 8.3.0.

[00:05:08.36] If I switch back to the Benchmark Factory console, I can see that I’ve got one job here, a TPC-C job. I can go back to Postman and get a list of my Benchmark Factory jobs. In this case, I’m also running a GET to localhost:30100/API. But this time, the endpoint is slash jobs. So I go ahead and send that command over. I get the response, the TPC-C job and all the properties of that job.

[00:05:46.52] In my Benchmark Factory, I have a list of connections. Three connections listed here. I’ll go ahead and say Close. Go back to Postman, and say get a list of my connections. In this case, again, it’s a GET, localhost, port number, now I’m doing connections. Go ahead and send that command over. And I get each of my connections. So we can get information from Benchmark Factory.

[00:06:31.55] Let’s now post some information to Benchmark Factory. So let’s post a new connection. So this time, I’m using a post command rather than a GET. I’m going to use the same host name and port number. And again, I’m going to point to the connection’s API. But this time, as part of my request, I’m going to send the body of a connection called Oracle_OraclePDB_PM, for Postman. And everything else is formatted in the same JSON format.

[00:07:16.89] And go ahead, and click on the Send button. And I get it a 201 Created. If I go over to Benchmark Factory and click on my connections, I can see that the connection has been created. So I can control Benchmark Factory entirely through this REST API. And again, you can find the documentation back on the website. For connections, get connections, post a connection, and the example of the JSON screen.

The Delete command

[00:08:05.22] So let’s take a look at the Delete command. You would just say delete connections and connection name. So let’s go back to Postman. Take a look at the delete connections, here. So I’m going to say delete, now, localhost, port name, once again. And this time, I have the name of my connection as part of this. I don’t need the body, actually. I could even say none, here. And say Send. I get a 204 No Content. But if I go back to Benchmark Factory, I can see the connection has been deleted.

Postman: Add a new job

[00:08:59.78] Now, let’s go back to Postman and add a new job. Going to post a job, here. And go to Body, and I’ve already filled this out prior. I’m going to post a job, called TPC-C Job from Postman. And 201 Created. And you can see here, Benchmark Factory has a new job from Postman.

[00:09:38.19] Now that we’ve created a job, let’s go ahead and run it and create from scratch, here. So let’s first get the status of this job. Slash API slash jobs. And the name of my job, which is TPC-C Job from Postman slash Status. My current status is 1. But to run it, you would change this to a PUT. And then change job, here. And give it a Body. Let’s copy this and put it through raw. Paste this in. It will change the status to 2.

[00:10:36.00] We’re going to Put, at this address, for this job, for the body, change the status to 2. I’m going to send that. And you can see Benchmark Factory is now running my job. It’s kicking off my agents. And when this is done, I’ll show you the results, in just a moment.

[00:11:13.80] So my job is completed. I ran 4 user loads, 1, 4, 8, and 10 users. Each load ran for about 10 minutes. And I could see for 10 users running my load, the transactions per second are about 0.52. And the response time of the database, about 3 seconds. It just goes to show that small loads, like 1 and 4 users, isn’t going to give you a good idea of how well your database will really perform. You really have to crank up the user load.

[00:11:47.30] So in this video, we talked about Benchmark Factory, what it is, and how it’s useful. We learned that you can access Benchmark Factory through its REST API to create jobs, run jobs, and get test results. In thenext video, we’ll use what we learned here and create scripts that can invoke the REST API. Then we could take those scripts and put them into a CI tool, like Jenkins or Bamboo. So stay tuned for the next episode. See you then.