Lift controllers example

I’ve recently put together a basic Lift example, based on an e-Commerce theme. It contains:

  • Product listing
  • Basket
  • Checkout
  • Order confirmation

It shows the following techniques:

  • How to write forms and process the response
  • How to submit forms using Lift’s form.ajax helper – used to submit the first two parts of the checkout
  • Using a session variable – the basket
  • Logging with slf4s

You can get the code from Github:

https://github.com/hedleyproctor/amur-lift-ecommerce-example

If you don’t have a Git client installed, simply click on the “Zip” button to get the code as a zip file. Then change into the base directory and do the following:

  1. Type sbt to start the simple build tool. This will download the jar files needed by sbt itself.
  2. Once the sbt shell has started, type update to download the jar files needed by the application.
  3. Compile the code with compile.
  4. Start jetty with jetty-run.

The app will then be available on http://localhost:8080. You should be able to go to the product listing page, put a product in your basket, then go to the checkout, enter your delivery address, choose a shipping option, enter your billing details and be taken to the order confirmation page. Click the images to enlarge:


Note:If you are very new to Lift and want an even simpler example, with a more detailed tutorial on how it works, you might like to look at Building your first Lift app with sbt.

Posted in Lift, Scala | Tagged , | Leave a comment

Effective Selenium testing

Selenium is the de facto standard for testing web applications. In this article I’m going to cover a number of techniques for improving your Selenium tests. The article is suitable for you if:
  1. You already know the basics of Selenium and are happy to write tests in Java (or one of the other Selenium API languages).
  2. You want to progress from having a few simple tests to building a large test suite for a complex web application with javascript and ajax.
The techniques I’ll cover are:
  1. Organising your tests – using the page controller design pattern
  2. Understanding the Selenum 2 webdriver and implicit waits
  3. Detecting when web elements are available
  4. Knowing when ajax calls have finished
  5. Taking screenshots for failing tests
The code snippets are in Java, but the Selenium API is available for a wide range of other languages (Ruby, Python, C#, PHP and Perl).

Organising your tests – using the page controller design pattern

If you write a large number of tests, when the application under test changes, you could face a maintenance nightmare. How do you avoid this? The answer is the page controller design pattern. This means that you have a Java class for each page of the application. It knows how to control that web page. i.e. it knows how to identify the elements on the page, how to select them etc. Your test classes don’t know anything about the web page. They don’t include any html ids or other selectors, they simply invoke methods on your page controller. If that web page changes, you only need to update your code in a single place. Code that used to look like this:
driver.findElement(By.id("checkoutButtonCartPage")).click();
driver.findElement(By.id("homeDeliveryButton")).click();
driver.findElement(By.id("postCodeEntry")).sendKeys("AB12 3CD");
driver.findElement(By.id("postCodeSubmitButton")).click();
becomes:
cartPageController.goToCheckout();
checkoutPageController.selectHomeDelivery();
checkoutPageController.postCodeLookup("AB12 3CD");

Understanding the Selenium 2 webdriver and implicit waits

One of the tricky aspects of Selenium testing is knowing when pages have loaded and when elements have appeared on pages. In Selenium 2 the webdriver includes some helpful functionality in this area, but it is important to understand what it does and doesn’t do. When you call the findElement method on the webdriver, if it cannot find the element on the page, it doesn’t fail immediately. Rather, it polls the page every 500ms until it reaches its timeout period. You can set this timeout period by calling webdriver.manage().timeouts().implicitlyWait. However, misunderstanding the webdriver polling functionality can cause confusion. The web driver findElement method returns a web element as soon as it finds it in the browser DOM. However, that doesn’t necessarily mean that you can interact with that web element. All elements have a display property. If the current CSS rules are not displaying that element, you can’t interact with it. The web driver won’t time out, rather it will find the element and return it to your code, which will promptly fail with an exception that explains you can’t interact with the element. A common example of this is a web page where pressing a button triggers a form to appear. Usually this form will already be in the web page, it is just hidden by CSS. Hence, if your Selenium test presses the button to make the form appear, if it proceeds to try and interact with the form too quickly, the CSS won’t have been switched over to display the form by the time the web driver locates it, and your test will fail. Thankfully, Selenium has additional functionality that can help us, which I’ll explain in the next section.

Detecting when web elements are available

We’ve just seen that it isn’t enough for a web element to be present in the browser DOM for us to interact with it, it needs to be visible as well. How do we detect this with Selenium? Well, the web driver allows you to poll for a specific condition to become true, by using the WebDriverWait class. It includes a number of standard conditions, of which visibility is one. This makes it easy to code up a helper method that will only return an element when it is visible:
public WebElement getWhenVisible(By locator, int timeout) {
	WebElement element = null;
	WebDriverWait wait = new WebDriverWait(driver, timeout);
	element = wait.until(ExpectedConditions.visibilityOfElementLocated(locator));
	return element;
}
Is this enough? Well, not necessarily. Interactive web elements also have an “enabled” property. e.g. if you want to show a checkbox but not allow the user to be able to change it, you set enabled to false. If you want to be certain that you can click an element, Selenium has another standard condition for this which you can make use of. e.g.
public void clickWhenReady(By locator, int timeout) {
	WebDriverWait wait = new WebDriverWait(driver, timeout);
	WebElement element = wait.until(ExpectedConditions.elementToBeClickable(locator));
	element.click();
}
For the full range of expected conditions, refer to the Selenium javadoc: ExpectedConditions

Knowing when Ajax calls have finished

If your test triggers an ajax call, you don’t want to carry on until that call has finished, but how do you know? You might be okay just to use one of the wait conditions above, but this isn’t a very clean approach, and it only works for ajax calls that result in changes to the browser DOM. i.e. it won’t work for calls that simply send data to the server without any changes in the browser html. Wouldn’t it be nice to be more certain when the call was finished? Well, if you are using jQuery to make your ajax calls, you can do so by exploiting the fact that most web driver implementations can run javascript. jQuery keeps a count of how many ajax calls are active in its jquery.active variable. Here’s an example of a helper method to wait for an ajax call to finish:
public void waitForAjax(int timeoutInSeconds)  {
  System.out.println("Checking active ajax calls by calling jquery.active");
    try {
      if (driver instanceof JavascriptExecutor) {
	JavascriptExecutor jsDriver = (JavascriptExecutor)driver;
				
        for (int i = 0; i< timeoutInSeconds; i++) 
        {
	    Object numberOfAjaxConnections = jsDriver.executeScript("return jQuery.active");
	    // return should be a number
	    if (numberOfAjaxConnections instanceof Long) {
	        Long n = (Long)numberOfAjaxConnections;
	        System.out.println("Number of active jquery ajax calls: " + n);
	        if (n.longValue() == 0L)
	       	  break;
	        }
            Thread.sleep(1000);
	    }
	}
	else {
		System.out.println("Web driver: " + driver + " cannot execute javascript");
	}
}
	catch (InterruptedException e) {
		System.out.println(e);
	}
}

Of course, this example could be rewritten to use the WebDriverWait format if you wish.

Taking screenshots for failing tests

One of the golden rules of good testing is that you should be able to diagnose why a test failure has occurred without rerunning the test. For Selenium, as well as having good assertions and debug output within the tests, it is very useful to take a screenshot when a failure occurs. If you are using JUnit for your tests, a neat way of doing this is to use a JUnit rule to take the screenshot. A good write up of how to do this is here:

http://blogs.steeplesoft.com/2012/01/grabbing-screenshots-of-failed-selenium-tests/

Summary

In this article we've seen:
  • How to organise your test suite by using the page controller design pattern
  • How the Selenium web driver works and how to write tests for web applications with javascript and ajax
  • How to take screenhots for failing tests
For more information about the web driver and testing design patterns, see the Selenium docs: http://seleniumhq.org/docs/index.html If you're interested in using CruiseControl to automate your tests: Automating Selenium testing with TestNG, Ant and CruiseControl If you'd like to learn about using XPath for complex element location: Writing XPath selectors for Selenium tests
Posted in Selenium, Testing, Uncategorized | Tagged , | 1 Comment

Why Java developers should be learning Scala

Over the past fifteen years Java has been a phenomenally popular programming language, but it is starting to show its age and programmers are increasingly looking at more modern languages. The purpose of this article is to explain why Scala is the most likely successor to Java and how it can make you more productive. Rather than simply listing the features that Scala has, I’ve included a number of comparisons between Java and Scala code, to demonstrate how the different Scala language features enable you to implement the same functionality more quickly in Scala than Java. Since Scala compiles to Java bytecode and runs on the JVM, programs written in Scala can benefit from the huge amount of library code already written in Java. However, by using Scala you get the following benefits:
  • Mandatory boilerplate code is gone – no getters and setters, no checked exceptions.
  • More powerful constructs, that allow you to do more with less code, such as case classes, option and tuples.
  • More powerful code reuse – the elements of code that you can reuse are smaller. Rather than classes with single inheritance, you have traits and functions.
I’ll work through these points in turn, giving examples of each.

No getters and setters

In Java, a class to represent a person with a name and age might be:
public class Person{
	private String firstName = null;
	private String lastName = null;
	private int age = 0;
	
	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}
	
	public String getFirstName() {
		return firstName;
	}
	
	public void setLastName(String lastName) {
		this.lastName = lastName;
	}
	
	public String getLastName() {
		return lastName;
	}
}
In Scala, most likely you would write this class as:
public class Person {
	var firstName = ""
	var lastName = ""
	var age = 0
}

The variables in this class are public. If you come from a Java background this sounds worrying - doesn't it mean that if we ever need getters and setters with additional code in them we'll have to change all of the code that uses this class? Not in Scala. This is because Scala has a very flexible method syntax, so we can write a method that looks the same as accessing the variables directly. An example would be:
public class Person {
	var firstName = ""
	var lastName = ""
	private var theAge = 0
	
	def age = theAge
	
	def age_= (newAge : Integer) : Unit = {
		if (newAge > 0) theAge = newAge
	}
}

In this example, I've renamed the variable to theAge and made it private, but written a getter and setter. The getter method is called age so you can write p.age to get the age, just like before. The setter is called age_=. The underscore has a special meaning here - it allows you to write a method name with a space in it. This means that when you write:
val p = new Person()
p.age = 33
you are actually invoking the new setter method.

No checked exceptions

When Java was invented, it seemed like a good idea to force developers to deal with certain possible error conditions, which led to the concept of checked exceptions. Scala has removed these. If you want to catch an exception, you can do so, but you're not forced to insert try/catch statements throughout your code if you don't want to.

Case classes

Case classes are like an enhanced version of the Java switch statement. They are small classes that are usually defined in the same class file as the real classes that you want to match. Unlike switch, they can understand different object types and extract data from them. Consider a scenario in which you are iterating over a tree structure that represents an organisation chart for a company. The nodes in the tree are either of type Group or Employee. If you find a Group node, you want to print out the name of the group and the size. If you find an employee, you want to print out their name and job title. In Java, your code would look something like: if (node instanceof Group) { Group g = (Group)node; System.out.println("Group name: " + g.getName() + " Size: " + g.getSize()); } else if (node instanceof Employee) { Employee e = (Employee)node; System.out.println("Name: " + e.getName() + " Job: " + e.getJob()); } In Scala this would be:
match node {
	case g: Group => println("Group name: " + g.name + " Size: " + g.size)
	case e: Employee => println("Name: " + e.name + " Job: " + e.job)
}

In this example I've used a "typed pattern" match, which avoids the type casts required in Java. If this was the only thing pattern matching could do, it wouldn't be that impressive, but it can do much more. There are several different sorts of pattern matching, the most powerful of which is probably a "constructor pattern". By matching against the contructor for a class, you can nest additional pattern matches against the values that have been passed into that constructor. These patterns can themselves be constructor matches, allowing you to match as deeply as you want. Continuing the example above, suppose that in addition to a manager, some groups also have a project manager. You want to find all groups that have a manager who is in salary band 10 and a project manager who is in salary band 9. In Java you'll need something like:

if (node instanceof Group) {
	Group g = (Group)node;
	Manager m = g.getManager();
	ProjectManager pm = g.getProjectManager();
	if (m.getJobBand() == 10 && pm != null && pm.getJobBand() == 9) {
		System.out.println("Group: " + g.getName());
	}
}

In Scala, with the appropriate case classes, this would be:
match node {
  case g: Group(Manager(10),ProjectManager(9)) 
    => println("Group: " + g.getName())
}



Option

In Java, it can be painful having to perform a != null check each time you get a variable that might be null. For example: items = shoppingBasket.getItems(); if (items != null) { for (Item i : items) { // process each item } } else { System.out.println("No items in shopping basket.") } The Scala standard library provides a class called Option, which has two subclasses, Some and None. Some is a container that wraps whatever class you are using. The basic pattern is that methods that could return null in Java return an Option, which will be either Some or None. Then calling code can use pattern matching on the returned value:
i = shoppingBasket.items
match i {
	case Some(items) => items.foreach( // process each item )
	case None => println("No items in shopping basket")
}

With the Java code, you can forget to insert the != null check, which can then lead to a NullPointerException at runtime, but with the Scala code, this isn't possible.

Tuples

How many times have you written a method in Java, only to find that you really want to return two things from the method, not one? In Java the standard way to fix this is to create a small class that just contains the return values, but then you are bloating your code by having a class when all you really need to do is specify that the method returns multiple things. Scala has exactly this concept with tuples. A tuple is common in functional languages and is simply a heterogenous list. It is written using brackets, so a tuple composed of the integer 5 and string "hello" would be written: (5,"hello") If you want to return multiple values from a method, you simply pass them back as a tuple like this.

Traits

In an effort to avoid the problems of multiple inheritance as it was defined in C++, Java eschewed multiple inheritance entirely. This can make reusing code from two places very difficult. In Scala you can only inherit from a single class but you can also mixin as many "traits" as you want. A "trait" can be thought of as similar to an abstract class in Java.
class MyQueue extends BasicIntQueue with Incrementing with Filtering
In C++, the above sort of statement could result in ambiguity as to which method to invoke. If Incrementing and Filtering both inherit from the same base class A, you have the "diamond problem" whereby there are two instances of class A. C++ addresses this by giving you the virtual keyword which ensures that there is only a single instance of A. In Scala, traits can extend other traits or classes, but Scala always has a defined order in which methods must be invoked, by using a linearization algorithm (similar to other languages such as Python). This means you get the code reuse benefits of inheriting from multiple places without the problems caused by non-virtual inheritance.

Functions as closures

In Java, if you want to allow callers to pass code into your class to be invoked, you have create an interface or concrete class before writing a callback method. Suppose that you have a class which contains a collection of Person objects. You want to write a method that will iterate over all of the Person objects and run some code that has been passed in, which will produce a summary of each Person as a String. In Java you would first have to declare an interface: public interface PersonSummariser { public String summarise(Person p); } Then you can write your callback method, specifying that code to be passed in must implement this interface: public void summarisePeople(PersonSummariser summariser) You've been forced to write an interface, and the person using your class has been forced to create a class (at best they might be able to create an anonymous class so they don't need a full class definition), just to pass in code that could be as short as a single line. In Scala, this would be handled by a closure. In mathematical terms, a closure is a function for which all of the variables are bound. i.e. given values. If the method you are writing supplies values for all of the parameters in the function that is passed in, you have a closure. The Scala method definition would be:
def summarisePeople(s : Person => String) 
Here we have written a method that accepts a function s, which takes a single parameter of type Person, and returns a value of type String. No need to create any additional interfaces or classes.

Closures are used extensively to perform operations on collections in functional languages. Here are just a few of the methods Scala provides in its collection classes which allow you to pass in a function to perform various operations:
  • map - transform a collection of type A to another collection of type A
  • filter - reduce the collection by filtering out all elements that don't meet a boolean condition
  • foldLeft - apply a function to each element of the collection in turn and sum the results e.g. square every integer in a list

Standalone functions

In the example above, if you declare the Scala summarise function inline, you are creating a closure. But functions are first class entities in Scala, so you can define them independently and reuse them wherever you want. Suppose you had multiple classes holding Person objects, such OrganisationChart, Company, Team and so on, if you wanted to define a function to print out the Person objects that you could pass into any method with the same signature, you could do so, anywhere in your code:
def summarise(p : Person) : String = p.firstName + " " + p.lastName
In fact, you don't need to declare that the return type on the above method is String, as the Scala compiler will infer it, but I added it for clarity. No longer is a class the smallest element of reuse you have, you can define individual functions and pass them around as you wish. 

Currying

Suppose you're writing code that calculates economic statistics for countries. You have a function that takes a population size and a GDP value. What if you wanted to invoke this multiple times with a fixed population size but differing GDP values? You might expect to have to repeat the first argument whenever you use the function:
calculateStats(pop1, 2000)
calculateStats(pop1, 10000)
In fact, you can "curry" the function, which means creating a new version of the function in which all but one of the parameters have already been supplied:
val cs = calculateStats(pop1, _ : Double)
Here we have supplied a value for the first parameter, but used the underscore to show that we're not supplying a value for the second parameter. You can then invoke this new "cs" method to calculate statistics specifically for countries of a specified size. At first this might not seem that powerful - surely we're just saving ourselves a bit of typing? However, consider that in Scala, a method parameter doesn't have to be a simple object, it can itself be a function. This makes it very easy to write code that is both powerful and flexible. You can write functions that perform specific tasks and combine them however you want.

Summary

If you haven't used Scala before, hopefully this article has persuaded you that it's worth investigating. We've seen that:
  • It doesn't require all of the boilerplate code that is needed in Java, such as getters, setters and checked exceptions.
  • It has powerful constructs that allow you to do more with less code, such as case classes and tuples.
  • It gives you better code reuse with traits and functions.
It's worthwhile explaining why I haven't mentioned a couple of things that Scala is known for - actors and parser combinators. Actors are a powerful mechanism for multi-threaded programming that avoid some of the problems of locks. Parser combination is a way of writing language parsers by combining lots of small parsers, rather than writing (or more likely generating) a single parser from a BNF grammar. Whilst both of these topics are interesting, I'm not sure either of them is necessarily indicative of the power of the Scala language. Each of them can be implemented in Java using an appropriate library - Kilim for actors and jparsec for parser combinators. By contrast, the topics I've covered above show things that have to be implemented within a language itself and cannot be provided by library code.

Hang on - what about languages like Ruby, Groovy or Clojure?

All of these languages are good, powerful languages that can make you more productive. However, you can't necessarily learn all four. Why should you choose Scala over the others? The feature set of Ruby, Groovy and Scala is broadly the same. They have all done away with getters and setters, and checked exceptions. They all have more functional concepts than Java, such as closures and first class functions. They all offer multiple inheritance via traits (mixins). However, both Ruby and Groovy are scripting languages that are dynamically and weakly typed so whilst they are good for small tasks such as automation, they don't lend themselves to constructing large enterprise applications as well as Scala and Clojure. Scala has a very powerful type system and compiler so many bugs can be found at compile time. Scala is a hybrid of object oriented and functional concepts, so its syntax is broadly object oriented, whereas Clojure is a lisp variant and hence uses Church's lambda calculus notation, which is a very different syntax. Finally, Scala does have some concepts which don't really appear in the other languages, of which the most obvious example is case classes, which offer a very powerful syntax for matching objects by type and extracting data from them.

Okay, you got me. How do I find out more about Scala?

If you want a comprehensive overview of the entire language, the first edition of the book "Programming in Scala" book is available free online: Programming in Scala In particular, some of the topics I've mentioned above are: Traits Case classes The Eclipse Scala IDE is available from: http://scala-ide.org/ Daniel Spiewak's blog has numerous good posts on Scala, such as: Funtional currying in Scala The Option pattern
Posted in Scala | Tagged | 4 Comments

Generating Javascript with Scala and Lift

One of the ideas that has become popular in recent years is the concept of writing complex browser GUIs entirely in your server side language, and generating the required javascript. In Java both the Google Web Toolkit and ZK allow you to do this. You get a number of advantages from this approach:
  • Your server side language is at a higher level of abstraction than coding in javascript and hence more productive.
  • You can get some type checking done when your code is compiled.
  • You can reuse functions you’ve already written as part of your core application, and the framework will translate them to javascript as required, rather than having to rewrite them in javascript yourself.
In this article I’ll introduce the functionality that the Scala Lift framework offers for generating javascript from Scala. I’m not assuming any prior knowledge of the Lift framework so you should be able to work through the examples even if you’ve never used Lift before. However, if you’d prefer to learn the basics of Lift and sbt first, you might want to read my previous tutorial: Building your first Lift app with sbt.

Download Lift

If you’re new to Lift, download it from: http://liftweb.net/download As the instructions say, if you change into the scala_28/lift_basic directory and run sbt update ~jetty-run then an empty Lift application should start.

Add a form and validation Javascript

To begin with, we’ll add a form to the main page of the app and attach some javascript to it. Open up the src/webapp/index.html and add a simple form into the body of the page:
<div class="lift:RegistrationController?form=post">
  First name: <input id="first_name"><br>
  Surname: <input id="last_name"><br>
  E-mail: <input id="email"></br>
  <input type="submit" value="Submit">
</div>
Now we need to create the snippet that will process this form. The class on the form indicates that it will be processed by a snippet called “RegistrationController” so go into src/main/scala/code/snippet and create RegistrationController.scala:
package code.snippet
import scala.xml.NodeSeq
import net.liftweb.util._
import Helpers._
import net.liftweb.http._
import net.liftweb.http.js.JsCmds._
import net.liftweb.http.js.JE._

class RegistrationController  {
	
	private val whence = S.referer openOr "/"
	
	def render = {
	  "type=submit" #> SHtml.submit("Register", process, 
	    "onclick" -> JsIf(JsEq(ValById("first_name"), ""), Alert("alert") & JsReturn(false)).toJsCmd)
	}
	
	private def process() = {
	  S.redirectTo(whence)
	}

}

Let's break down what this snippet is doing. As with all snippets, it implements the render method to transform the html in the page template. It is using a Lift CSS selector transform (the #>) which finds the html element with the "type" attribute set to "submit", and replaces this with a submit button generated by calling the SHtml.submit method. The first parameter to the submit method is the name of the button, the second is the function to be called when the form is sent to the server. The third parameter is a list of html attributes that the button element should have. In this case we are just creating a single attribute, the onclick one, which will invoke our javascript validation - if the first_name parameter is empty, show a pop up alert box. We call the toJsCmd method on the javascript expression to turn it into a command, because that can be implicitly converted into the html attribute that the submit method requires. You should be able to verify that this alert is shown if you try to submit the form without a first name.

Let's extend the example slightly to show how we can edit the browser DOM. Let's add a second button to the html template next to the first: First name:
Surname:
E-mail:
In our Scala code, we'll add some javascript to this button to populate the first name field with a default value when you click the button:

def render = {
  "type=submit" #> SHtml.submit("Register", process, 
     "onclick" -> JsIf(JsEq(ValById("first_name"), ""), Alert("alert") & JsReturn(false)).toJsCmd) &
  "type=button" #> SHtml.ajaxButton("Populate form", () => JsCmds.SetValById("first_name","John"))
}

What is this Scala doing? Well, we begin by using an & to join our chain our first CSS selector to a second one. The second CSS selector finds the html element with the "type" set to "button" and replaces it with a button created by calling the SHtml.ajaxButton method. This method takes two parameters, the first is the text to put on the button, and the second is a function to invoke when the button is pressed. We use standard Scala syntax of => to declare a function that takes no parameters and when invoked, finds the first_name element in the DOM and sets the value to "John". 

Wrap up

In this article we've looked at a couple of simple examples of how to generate javascript on the server, using Scala code. The basic mechanism is to create the javascript using Lift's JsCmds and JE classes, then attach the javascript to your html using standard CSS selectors and the helper methods in the SHtml object. For more detailed examples, you can check out chapters 10 and 11 in the "Exploring Lift" book: http://exploring.liftweb.net/master/index-10.html
Posted in Javascript, Lift, Scala | Tagged , , | Leave a comment

Using Java to download a file that needs authentication

You can easily download files using Java by making a URLConnection. However, if you need to login before accessing the file, how can do this automatically? If you automate the login, how will the code that makes the URLConnection be able to make use of the logged in session? Well, if the logged in session uses a cookie, you can simply extract the cookie and resend it in your code that makes the URLConnection.  Here is an example where I’ve used the Selenium web testing tool to do the login. The code is actually Scala, but you can easily convert it to Java:
  def getFileViaSelenium() {
    println("Logging in via Selenium")
	  val driver = new FirefoxDriver()
	  driver.get("https://www.someurl.com/login")
	  driver.findElement(By.id("username")).clear();
	  driver.findElement(By.id("username")).sendKeys("John Smith");
	  driver.findElement(By.id("password")).clear();
	  driver.findElement(By.id("password")).sendKeys("password");
	  driver.findElement(By.name("commit")).click();
 // now get the cookies	 
   val seleniumCookies = driver.manage().getCookies().asScala
	  val cookieString = new StringBuilder()
	  for (cookie <- seleniumCookies) {
	    println("Cookie value: " + cookie.getValue())
	    cookieString.append(cookie.getName())
	    cookieString.append("=")
	    cookieString.append(cookie.getValue())
	    cookieString.append("; ")
	  }
	    
     println("Getting file")
    
    val url = new URL("https://www.someurl.com/somefile.csv")
    val con = url.openConnection()
  // resend the cookies with this request
    con.setRequestProperty("Cookie",cookieString.toString())
    val contentType = con.getContentType()
    println("Content type: " + contentType)
    val in = con.getInputStream()
    // save file
    val fileOut = new File("C:\my_download.csv")
    inputToFile(in,fileOut)
    println("File saved")
  }
							
Posted in Java, Scala, Selenium | Tagged , , | Leave a comment

Organising Eclipse static imports

By default Eclipse “Organise imports” doesn’t deal with static imports, so if you are using JUnit and want to write something like assertEquals(), it won’t be imported. However, you can add static imports to your Java preferences to get them available via the quick assist (CTRL+1). Details on stack overflow:

http://stackoverflow.com/questions/288861/eclipse-optimize-imports-to-include-static-imports

Posted in Eclipse, Java | Tagged , | Leave a comment

Getting the last build date from CruiseControl

Imagine you have a part of your build process that is a time consuming operation that only needs to take place when a particular file has been updated. How would you get the time of the last successful build? Well, CruiseControl stores a log file for each build, so one way to do it is to list the contents of the log directory and locate the latest file. Here is the Ant syntax:
   
     
       
       
     
     
   


Obviously this uses a shell command so it will only run on unix / linux. You need to be careful when passing a sequence of pipe joined commands to the shell from Ant. You can't use the name of the first command as the executable, as the remainder of the command will be fed in as a parameter, which isn't correct. Instead, simply specify the shell as the executable and use -c to specify the full command line that you want to execute. The breakdown of the above command is:

  • Find all of the log files which start with log and have the word 'build' in. Unsuccessful builds will have a log file, but it won't have the word 'build' in. In the example above, I'm assuming that your ant script is in [CRUISE_CONTROL_ROOT]projectsprojectname and the logs are in [CRUISE_CONTROL_ROOT]logsprojectname, so the find command has to recurse two directories before going into the log directory.
  • Sort the list of successful build files alphabetically.
  • Get the last file name - the latest build.
  • Use the sed stream editor to extract the latest build timestamp. Unlike a normal regex search and replace command, which removes elements of a string not matched by the regex, sed passes the entire string through, only replacing sections that it has matched. Hence to remove parts of a string you need to explicitly match them, then ensure that the output part of the sed command does not print them out. In the above example, the sed command does the following:
    • .* - match the first part of the file name, but don't capture the characters
    • ([0-9]{14}) - match the 14 digit timestamp and store it in capturing group 1
    • .* - match the remainder of the file name, but don't capture the characters
    • Then the replace part of the sed command simply prints out the captured timestamp using 1
Posted in Ant | 2 Comments

Comparing two MySQL databases

When doing development, you’ll often have multiple versions of a database on different environments and sometimes it would be useful to be able to check the differences between the data. Did you know you can use the Toad database tool to compare two databases? Simply go to Tools -> Compare -> Data:

The next screen will ask you to map the tables from the first database to those in the second. If you are comparing databases with the same schema, you can simply select “Map all”. Then Toad will analyse the differences and give you a report which tells you how many rows are the same in each table, how many have been removed and how many added:

Posted in Databases and SQL, MySQL | Tagged | 2 Comments

Tutorial: Building your first Lift app with sbt

The Simple Build Tool (sbt) is the standard build technology for Scala and Lift applications. However, if you are new to Scala and Lift, trying to learn sbt at the same time can be a little confusing. The aim of this article is to give you enough knowledge about both sbt and Lift to be able to build and debug a simple Lift application using sbt 0.7. We’ll start with a “blank” Lift app and do the following:
  1. Run some sample sbt commands to become familiar with sbt.
  2. Examine the sbt directory structure.
  3. Look at the sbt LiftProject.scala configuration file that is used to build the blank Lift app.
  4. Examine Lift’s Boot.scala, web.xml and default.html that are used for all Lift apps.
  5. Add in the Lift chat server classes from David Pollack’s example and rebuild and retest the project.
No prior knowledge of Lift, sbt, or even Scala is assumed. This tutorial uses sbt 0.7 because that is what Lift currently uses. However, sbt is evolving fast and version 0.10 has a number of key differences. I’ll try and point these out in the tutorial so that you don’t get caught out if you decide to play around with a later version of sbt.

Downloading Lift

You can download Lift from:

http://liftweb.net/download

The version I’m using for this tutorial is 2.4-M4. If you open up the Lift directory, you’ll see it has two subdirectories, one called “scala_28” and one called “scala_29”. Each of these directories contains four templates:
  • lift_basic
  • lift_blank
  • lift_mvc
  • lift_xhtml
We’ll use the Scala 2.8 lift_blank template, so take a copy of the directory and rename it. On my system it is c:lift-chat-tutorial.

Basic sbt commands

To get familiar with sbt, you can try a few a simple commands. Change to the directory you have just created and start up an interactive command prompt by typing “sbt”. sbt is self-bootstrapping, so what you are actually doing is running a small piece of launch code which will download the main sbt code using Apache Ivy. You should be able to see the main parts of sbt being downloaded:
C:lift-chat-tutorial&gt;sbt

C:lift-chat-tutorial&gt;set SCRIPT_DIR=C:lift-chat-tutorial

C:lift-chat-tutorial>java -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m 
-Xmx512M -Xss2M -jar "C:lift-chat-tutorialsbt-launcher.jar"
Getting Scala 2.7.7 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
        confs: [default]
        2 artifacts copied, 0 already retrieved (9911kB/264ms)
Getting org.scala-tools.sbt sbt_2.7.7 0.7.5 ...
:: retrieving :: org.scala-tools.sbt#boot-app
        confs: [default]
        16 artifacts copied, 0 already retrieved (4271kB/594ms)
[info] Recompiling project definition...
[info]    Source analysis: 1 new/modified, 0 indirectly invalidated, 0 removed.
Getting Scala 2.8.1 ...
:: retrieving :: org.scala-tools.sbt#boot-scala
        confs: [default]
        2 artifacts copied, 0 already retrieved (15118kB/302ms)
[info] Building project Lift SBT Template 0.1 against Scala 2.8.1
[info]    using LiftProject with sbt 0.7.5 and Scala 2.7.7
sbt uses Apache Ivy to manage dependencies, so downloaded files are put in a local Ivy cache. By default this will be in a directory called .ivy2 in your home directory, so on my Windows 7 machine, they are in:
C:Usershedley.proctor.ivy2
You should be able to browse to this directory on your system and see the jar files that sbt has downloaded.

Type “help” to see the list of available sbt commands. At first glance it may look like a short list. This is because in sbt 0.7, most sbt commands are called “actions”, so to see the available actions, type “actions”:

&gt;actions
        -empty
        clean: Deletes all generated files (the target directory).
        clean-cache: Deletes the cache of artifacts downloaded for automatically...
        clean-lib: Deletes the managed library directory.
        clean-plugins
        compile: Compiles main sources.
        console: Starts the Scala interpreter with the project classes on the classpath.
        console-quick: Starts the Scala interpreter with the project classes on the...
        copy-resources: Copies resources to the target directory where they...
        copy-test-resources: Copies test resources to the target directory...
From version 0.10 of sbt, the nomenclature has changed, so that everything is a “task”. Hence to see in the commands in sbt 0.10 you would type “tasks”.

The “current” command will print the details of the current project:

&gt;current
Current project is Lift SBT Template 0.1
Current Scala version is 2.8.1
Current log level is info
Stack traces are enabled
From sbt 0.10, you use the “project” command to see the current project information. Let’s build this blank app and check it works. The first thing to do is ask sbt to check the dependencies of the project and use Apache Ivy to download any that are missing. You can do this using the “update” command:
&gt; update
[info]
[info] == update ==
[info] :: retrieving :: Lift#lift-sbt-template_2.8.1 [sync]
[info]  confs: [compile, runtime, test, provided, system, optional, sources, javadoc]
[info]  21 artifacts copied, 0 already retrieved (14247kB/835ms)
[info] == update ==
[success] Successful.
[info]
[info] Total time: 11 s, completed 13-Oct-2011 21:00:45
Now you can start the application with “~jetty-run”. Although you haven’t explicitly told sbt to compile the Scala code, jetty-run is dependent on the compile, so it will invoke it automatically. The ~ character tells sbt to recompile and republish the code if it changes, so by starting the command with ~, you can leave jetty running as you update the files.
&gt;jetty-run
[info]
[info] == copy-resources ==
[info] == copy-resources ==
[info]
[info] == compile ==
[info]   Source analysis: 3 new/modified, 0 indirectly invalidated, 0 removed.
[info] Compiling main sources...
[info] Compilation successful.
[info]   Post-analysis: 11 classes.
[info] == compile ==
[info]
[info] == prepare-webapp ==
[info] == prepare-webapp ==
[info]
[info] == jetty-run ==
2011-10-13 21:07:08.197:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
[info] jetty-6.1.26
[info] NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
[info] Started SelectChannelConnector@0.0.0.0:8080
[info] == jetty-run ==
[success] Successful.
[info]
[info] Total time: 8 s, completed 13-Oct-2011 21:07:09
You should now be able to go to your blank Lift app at:

http://localhost:8080

After you’ve experimented with the blank app, let’s take a look at how it is created.

sbt directory structure

sbt follows the Maven conventions for directory structure. In addition, Lift has some conventions of its own, so the structure of a Lift app is:
project/  
src/
  main/
    resources/
    scala/
      bootstrap/
      code/
        comet/
        lib/
        model/
        snippet/
        view/
    java/
    webapp/
  test/
    resources
    scala/
    java/
target/
The standard directories are:
  • project directory – contains the sbt configuration files to build the project
  • src/main – your application source
  • src/main/webapp – contains files to be copied directly into your web application, such as html templates and the web.xml
  • src/test – your test code
  • target – where your compiled application will be created and packaged

sbt configuration files

sbt has two methods of configuration – “light” and “full”. “light” configuration is done using files ending in .sbt that contain lists of settings. “full” configuration uses .scala files. Light configuration is a fairly new way of configuring an sbt project, so at the moment Lift uses full configuration. If you expand the project/build directory you can open the LiftProject.scala file that defines the build for this project. Since this is a simple project, all it really does is define some dependencies:
  override def libraryDependencies = Set(
    "net.liftweb" %% "lift-webkit" % liftVersion.value.toString % "compile",
    "org.mortbay.jetty" % "jetty" % "6.1.26" % "test",
    "junit" % "junit" % "4.7" % "test",
    "ch.qos.logback" % "logback-classic" % "0.9.26",
    "org.scala-tools.testing" %% "specs" % "1.6.6" % "test"
  ) ++ super.libraryDependencies
Now let’s look at the basic Lift files.

Basic Lift files

Lift always has a Boot.scala file in src/main/scala/bootstrap/liftweb which defines the global setup for the application. Open this file and take a look at it. The main things it does are:
  • Defines the packages for Lift code
  • Creates the sitemap – a menu and security access for the site
If you open webapp/WEB-INF/web.xml you can see the servlet filter that runs a Lift app:
&lt;filter&gt;
  &lt;filter-name&gt;LiftFilter&lt;/filter-name&gt;
  &lt;display-name&gt;Lift Filter&lt;/display-name&gt;
  &lt;description&gt;The Filter that intercepts lift calls&lt;/description&gt;
  &lt;filter-class&gt;net.liftweb.http.LiftFilter&lt;/filter-class&gt;
&lt;/filter&gt;  	

&lt;filter-mapping&gt;
  &lt;filter-name&gt;LiftFilter&lt;/filter-name&gt;
  &lt;url-pattern&gt;/*&lt;/url-pattern&gt;
&lt;/filter-mapping&gt;
You might also want to take a quick look at the default Lift html template, which is in webapp/templates-hidden/default.html.

Updating the application

We’re now ready to update the application to include David Pollack’s chat server code. The chat server example is given in chapter two of Simply Lift. There are four files:
  • View – should go in your webapp/index.html.
  • Chat.scala – should go in src/main/scala/code/comet.
  • ChatServer.scala – also in the code.comet package.
  • ChatIn.scala – this is the snippet that takes the input form data and submits it to the chatserver, so it goes in src/main/scala/code/snippets.
If you started jetty with ~jetty-run, sbt should detect the change in your source files and automatically recompile and republish the code, so you can return to your Lift app and type in messages which should appear on the web page.

Wrap up

In this tutorial you’ve learnt the following:
  • Basic sbt commands such as update, compile and jetty-run.
  • How sbt uses Maven conventions for directory structure.
  • That sbt projects can be defined using .sbt or .scala files which, at a minimum, will define the dependencies of the project.
  • How Lift uses a startup file called Boot.scala, a servlet filter called LiftFilter and a default.html template.

Links

Lift controllers example – a slightly larger example which shows you how to create forms, submit them using ajax and use session variables.
http://simply.liftweb.net – David Pollack’s book which covers the sitemap, snippets, forms, wiring, CSS transforms, RESTful web services and JSON.
https://github.com/harrah/xsbt/wiki – the sbt wiki has plenty of information, but remember that sbt has changed a lot between 0.7 and 0.10. Its useful to start reading up on current sbt usage, but if you do want information specifically for sbt 0.7, you are probably better off looking at the old Google project.
Posted in Lift, Scala | Tagged , | 3 Comments

Inserting large amounts of data into MySQL with LOAD DATA INFILE

Suppose you want to do some performance testing on your application and you want to insert a large number of database records, what is the fastest way to do it? You could write some Java code that creates a PreparedStatement and reruns it with different values of the parameters. However, MySQL provides a much faster way – using LOAD DATA INFILE to read in the data from a CSV file. Suppose you have a customer table with the following columns:
  • id
  • email
  • password
  • title
  • first_name
  • last_name
Say you’d like to insert 500,000 customers into this table. You can generate a CSV with some dummy customer data with a few lines of Java code:
import java.io.File;
import java.io.FileWriter;

public class CreateCustomerDataFile {
  public static void main(String[] args) throws Exception {
    final File f = new File("customers.csv");
    final FileWriter fw = new FileWriter(f);
    final String template = ""john.smithX@fake.com","password","Mr","John","Smith"n";
    final StringBuilder sb = new StringBuilder();
    for (int i=1; i<=500000; i++) {
      // make the e-mail unique by replacing john.smithX with john.smith1 etc
      // clear the StringBuilder
      sb.setLength(0);
      // initialise it to the template
      sb.append(template);
      // insert a unique number
      fw.write(sb.replace(11, 12, String.valueOf(i)).toString());
    }
    fw.flush();
    fw.close();
  }
}
 
By default the MySQL LOAD DATA INFILE command will load from the directory associated with the database that you are trying to load into, so once you have created the file, copy it to that location. e.g. on a Windows box, if the database you are using is called ecommerce_site, the directory will be something like C:Program FilesMySQLMySQL Server 5.0dataecommerce_site.

Then start up the MySQL command prompt with: mysql -uadmin -ppassword Once in the command prompt, you can read in the CSV with: load data infile 'customers.csv' into table ecommerce_site.customer fields terminated by ',' enclosed by '"' lines terminated by 'n' (email,password,title,first_name,last_name); In this example you'll need to specify the field separator as the default separator is a tab. For more info on the command, see:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html.

Posted in MySQL, Performance, Testing | Tagged , , | Leave a comment