AgileNorth - May 2010

On Friday I attended AgileNorth May 2010 at Preston UCLAN.

It was an excellent day and thankyou to all the organizers and sponsers Valtech, Thoughtworks to name two.

The opening keynote was by Kevin Murray of Valtech, titled 'Agile in Waterfall Case Studies'. Kevin talked about two case studies where he had been the project manager and introduced agile working practises. Both were large goverment waterfall projects with the latter being a project rescue.

The second project was especially interesting. He was the 14th PM in 2 years and the team members were desperately demotivated. Kevin identified the following benefits:

• increased morale


• increased motivation


• feeling of ownership


• issues addressed earlier


• visibility to all of status


• better quality


• less bugs at FAT

Kevin emphasised that introducing agile especially into large government style projects must not:

• compromize estimates


• question or sacrifice governance

The second talk I attended was by Nick Mckenna of Mckenna Consultants.
Nick is an agile consultant based in Harrogate offering training and consultancy and his talk was titled 'Agile Adoption'. The talk was good and interesting but felt to me like we were getting the first hour of a 1 day talk. His main point was to define a timetable when starting with agile and that could typically be about 3 months. He also emphasised the following:

• get everyone to agree


• identify roles


• spend some time on training


• take it slow


• prepare to compromise


• make it visible


• keep everyone informed


• the timetable must be agile


• review the time table

The third talk was by Paul Shannon (twitter) and Craig Judson (twitter) of Codeweavers titled 'From Chaos to Kanban'

Their talk discussed the life story of how Codeweavers had adopted and adapted agile over the last 2 or 3 years along with the help of the agile consultant Kevin Rutherford. They are clearly exstatic with where they are now in terms of performance and quality. They emphasised to me how agile is in itself agile. how they started doing scrum by the book and ended up with their own kanban solution and board. Their slides are available here.

An excellent lunch was followed by a workshop from Sam Wessel also of Codeweavers on 'CRC - the forgotten Design Tool'. Here is one link to CRC - Class Responsibility Collaborator. This was a fun workshop designing a library application. It showed how to really break down an issue and see your self as the objects in the problem along with their inter-reactions and roles.


The penultimate talk was by the Agile consultant Kevin Rutherford of Rutherford Software (twitter) entitled 'Flow'. The aim of the talk was to show how agile increases the speed with which changes can flow through from requirements to an end product. I really like Kevin's dry and straight talking style. His blog write up of the talk is here. Here are some of his points:

• YAGNI - do not do what you do not need now


• question or sacrifice governance


• thin sliced MMFs (minimal marketable feature) - IE breakdown the work into small chunks


• continuous integration


• Theory of constraints


• The Goal by Goldratt

The closing keynote - phew it was a long day - was by Jim Barritt and Mark Crossfield. Mark is a Thoughtworks ">Thoughtworks consultant and Jim works for Autotrader. There talk covered the issues of working running an agile product with a rapidly expanding workforce.

Two interesting tools that I came across in talks and will investigate soon are

1) Migratordotnet - This is based on the idea of Rails ActiveRecord Migrations. Interestingly it puts the migrations into a dll making them easy to run & install on a windows PC without the need for Rails or other bespoke software.

2) AgileZen - This is an online kanban board, free to use for a single project.

The next AgileNorth will be October time and will be free.

Find other links on AgileNorth via this delicious tag.

Finally here are some twitter links:
@BlueReZZ - Paul Shannon of Codeweavers
@craigjudson - Craig Judson of Codeweavers

SSL, tomcat and self signed certificates

I have been through the process of getting an application to work via https before but I have always muddled through and never documented exactly what I have done.

I must also admit that I never quite understood exactly what I had done and why.

One problem is that I can not find one place that covers everything you need in one place.

Now I think I am pretty well there.

This post describes what I have done to get a simple client application working using HTTPS posting to tomcat server and using a self signed certificate with the help of several refereneces in particular:

1. How to create a self signed certificate
2. Tomcat SSL Configuration How-To
3. Developing Web Services with Apache Axis by Tong Ka Iok, Kent ISBN 1-4116-7032-9

The last reference has a chapter which is pretty well SSL for dummies and explains things very nicely.

The steps that need to be done are:

1. Create a self signed certificate authority (CA)
2. Sign a test key via the CA
3. Add both these keys to a keystore
4. Setup the application (client) and tomcat (the server) to use this keystore.

1) Create a self signed certificate authority (CA) and keystore

This is described in How to create a self signed certificate, but I will show the steps here

What is happening here:

you will create a CA that later will be added to your keystore file. By adding this CA to your keystore you are saying it is trusted like verisign and any certificates signed by it are then also trusted.

1.1) make a directory to hold the certs and keystore. This might be something like:

C:\ssl

1.2) generate a private key for the server

openssl genrsa -des3 -out server.key 1024

1.3) generate a CSR (Certificate Signing Request)

openssl req -new -key server.key -out server.csr

1.4) Remove the passphrasse from the key

cp server.key server.key.org

openssl rsa -in server.key.org -out server.key

1.5) Generate the self signed certificate

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

2) Create a certificate for tomcat and add both to the keystore

2.1) cd to where the keystore is held. This might be something like:

C:\ssl

2.2) Create a keypair for 'tomcat'

keytool -genkey -alias tomcat  -keyalg RSA -keystore tomcat.ks

2.3) Generate a CSR (Certificate Signing Request) for tomcat

keytool -keystore tomcat.ks -alias tomcat -certreq -file tomcat.csr

2.4) create unique serial number

echo 02 > serial.txt

2.5) Sign the tomcat CSR

openssl x509 -CA server.crt -CAkey server.key -CAserial serial.txt -req -in tomcat.csr -out tomcat.cer -days 365

2.6) Import the server CA certificate into the keystore

keytool -import -alias serverCA -file server.crt -keystore tomcat.ks

2.7) add the tomcat certificate to the keystore

keytool -import -alias tomcat -file tomcat.cer -keystore tomcat.ks

3) Tomcat configuration

3.1) Tomcat needs to be configured to use SSL

This is described in more detail at Tomcat SSL Configuration How-To

However all that is needed here is to edit the server.xml to enable SSL

This section is already in the server.xml but commented out.

NB that the location of the keystore has been added.

<!-- Define a SSL HTTP/1.1 Connector on port 8443

This connector uses the JSSE configuration, when using APR, the

connector should be using the OpenSSL style configuration

described in the APR documentation -->

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"

maxThreads="150" scheme="https" secure="true"

keystoreFile="C:\ssl\tomcat.ks"

keystorePass="changeit"

clientAuth="false" sslProtocol="TLS" />

3.2) test tomcat

start tomcat and go to https://localhost:8443/

your browser will return an error such as "sites certificate is not trusted"

3.3) import the CA certificate server.crt into your browser's tructed root certificates

3.4) test again at https://localhost:8443/

this time you should see the tomcat home page

4) Test your application

4.1) I have a unit test run from eclipse that I have been using to post off to my test server. This produces the error;

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:174)

at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:238)

at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:289)

4.2) The reason for this is that eclipse is not referring to the keystore that we have created. At the time of writing I have not sussed out how to make eclipse use this - Do you know ?

So instead I have added the same two certs to javas keystore

4.3) cd to the directory where java's keystore is held. This might be something like:

C:\java\jdk1.6.0_18\jre\lib\security

4.4) the keystore is a file called cacerts

4.5) copy the files c:\ssl\server.crt and c:\ssl\tomcat.cer to this directory

4.6) import the server CA into the java keystore

keytool -import -alias serverCA -file server.crt -keystore cacerts

4.7 import tomcats cert into the java keystore

keytool -import -alias tomcat -file tomcat.cer -keystore cacerts

5) Test your app again.

Hopefully all will be hunkdory. Enjoy.

6) Caveats:

6.1) the passwords for all keystores and certs are 'changeit'. this is the default keystore password

and I suggest you change this for a production system

6.2) Using a self signed cert is great for a test environment of for a private system but not for a commercial released application. For this you will need to get & pay for a signed certificate from an approved authority such as Verisign.

train times using RESTful urls

While travelling by train in this arctic winter I came across this really nice simple train times site using RESTful URLs to great effect.

http://traintimes.org.uk/blackburn/entwsitle/16:00/today

The output is really clear and simple making it great for display on small phone screens.

Apache Ivy - Working with the Apache tutorial

Apache Ivy is a dependancy manager and appears to be flavour of the month in some circles and claims to be less complicated than Maven.

Here is a quick tutorial that expands on Apaches tutorial page. They provide one very small build.xml in a style that they say is not best practise for a number of reasons:

1. it is standalone
2. it contains the code within the build.xml
3. it downloads ivy locally instead of using a remote installation

I struggled to find a simple example of a build in the accepted syntax so I will attempt here to describe how this build.xml can be changed in 4 steps to ultimately work in the conventional style with a build.xml and ivy.xml files and a remote installation of ivy.

My next project will be to 'ivyfy' a spring project. Hopefully I will be successfull - watch this space.

prerequisites:

Apache Ant installed to 1.6 or greater

1) The basic Tutorial from Apache Ivy

• Essentially follow their instructions & download the build.xml from the tutorial page
• I had to edit the downloaded file to remove the dashes at the start of some lines
• run 'ant'
• I also changed the following line to refer to the latest formal release of ivy, but both will work.

from:

 <property name="ivy.install.version" value="2.0.0-beta1" /> 

to:

    <property name="ivy.install.version" value="2.1.0" />

2) Tutorial with no ivy

• One of things about ivy is that you are supposed to take an existing project and 'ivyfy' - IE remove your local third party jars and rely on ivy to fullfill your dependancies
• So this tutorial simply takes the results from the previous example and gets it to work without ivy.
• It therefore needs the src folder & its contents, the commons-lang jar in a lib dir in the project and a build.xml
• The file structure for this and future tutorials is:

.
./build.xml
./file
./filestructure
./lib
./lib/commons-lang-2.1.jar
./src
./src/example
./src/example/Hello.java

• The build.xml required is:

<project name="go-ivy" default="go" >

<property name="build.dir" value="build" />
<property name="src.dir" value="src" />
<property name="lib.path.id" value="lib" />

<path id="compile.classpath">
<fileset dir="${lib.path.id}">
 <include name="**/*.jar"/>
</fileset>
</path>

<!-- =================================
     target: go Go ivy, go!
    ================================= -->
<target name="go" depends=""
       description="--> resolve dependencies, compile and run the project">
   <echo message="NOT using ivy to resolve commons-lang 2.1..."/>

   <echo message="compiling..."/>
   <mkdir dir="${build.dir}" />
   <javac srcdir="${src.dir}" destdir="${build.dir}" >
      <classpath refid="compile.classpath"/>
   </javac>

   <java classname="example.Hello">
       <classpath>
          <fileset dir="${lib.path.id}">
            <include name="**/*.jar"/>
          </fileset>
          <path location="${build.dir}" />
       </classpath>
   </java>
</target>


<!-- =================================
     target: clean         
    ================================= -->
<target name="clean" description="--> clean the project">
   <delete includeemptydirs="true" quiet="true">
       <fileset dir="${src.dir}" />
       <fileset dir="${build.dir}" />
   </delete>
</target>
</project>

3) Tutorial with ivy.xml

• In the above example everything is defined in the build.xml. The clever thing with ivy and maven is that your project does not explicitly contain third party jars.
• Instead these are defined in a config file, ivy.xml in this case.
• This step moves the dependancies defined in build.xml into a seperate ivy.xml
• again simply run 'ant' to test.
• For this and subsequent tutorials I did not generate the src files. I kept them from the first tutorial
• Therefore for this and subsequent tutorials all you need is the src folder with the src file, the build.xml & ivy.xml
• Here is the ivy.xml

<ivy-module version="2.0">
<info organisation="Utilisoft" module="Test-withivy.xml"/>
<dependencies>
<dependency org="commons-lang" name="commons-lang" rev="2.1"/>
</dependencies>
</ivy-module>

• Here is the new build.xml - I have removed much of the comments to make it smaller.

<project name="go-ivy-with-ivy.xml" default="go" xmlns:ivy="antlib:org.apache.ivy.ant" basedir=".">
<!-- here is the version of ivy we will use. change this property to try a newer
version if you want -->
<property name="ivy.install.version" value="2.1.0" />
<property name="ivy.jar.dir" value="${basedir}/ivy" />
<property name="ivy.jar.file" value="${ivy.jar.dir}/ivy.jar" />

<property name = "ivy.lib.dir" value="${basedir}\newlib" />

<path id="lib.path.id">
  <fileset dir="${ivy.lib.dir}" />
</path>

<property name="build.dir" value="build" />
<property name="src.dir" value="src" />

<!-- =================================
target: go Go ivy, go!
================================= -->
<target name="go" depends="resolve "
description="--> resolve dependencies, compile and run the project">
<echo message="using ivy to resolve commons-lang 2.1 with ivy.xml"/>

<echo message="compiling..."/>
<mkdir dir="${build.dir}" />
<javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="lib.path.id" />

<java classname="example.Hello">
<classpath>
  <path refid="lib.path.id" />
  <path location="${build.dir}" />
</classpath>
</java>
</target>


<!-- =================================
target: clean
================================= -->
<target name="clean" description="--> clean the project">
<delete includeemptydirs="true" quiet="true">
<fileset dir="${src.dir}" />
<fileset dir="${build.dir}" />
</delete>
</target>

<target name="resolve" depends="install-ivy" description="Resolve the dependencies">
<ivy:retrieve/>
</target>


<!-- =================================
target: install-ivy
this target is not necessary if you put ivy.jar in your ant lib directory
if you already have ivy in your ant lib, you can simply remove this
target and the dependency the 'go' target has on it
================================= -->
<target name="install-ivy" depends="download-ivy" description="--> install ivy">
<!-- try to load ivy here from local ivy dir, in case the user has not already dropped
it into ant's lib dir (note that the latter copy will always take precedence).
We will not fail as long as local lib dir exists (it may be empty) and
ivy is in at least one of ant's lib dir or the local lib dir. -->
<path id="ivy.lib.path">
<fileset dir="${ivy.jar.dir}" includes="*.jar"/>
</path>
<taskdef resource="org/apache/ivy/ant/antlib.xml"
    uri="antlib:org.apache.ivy.ant" classpathref="ivy.lib.path"/>
</target>

<target name="download-ivy" unless="skip.download">
<mkdir dir="${ivy.jar.dir}"/>
<!-- download Ivy from web site so that it can be used even without any special installation -->
<echo message="installing ivy..."/>
<get src="http://repo1.maven.org/maven2/org/apache/ivy/ivy/${ivy.install.version}/ivy-${ivy.install.version}.jar" dest="${ivy.jar.file}" usetimestamp="true"/>
</target>
</project>

4) Tutorial with remote Ivy

• Hopefully you are still with me. This is the last step in this tutorial set.
• The above example still downloads ivy locally
• This example will now use ivy as part of your apache and installation
• In the last example a ivy.jar file appeared in an 'ivy' directory.
• Determine the location of your apache ant installation and copy this jar into the apache ant lib directory.
• Ensure you have an ANT_HOME environment variable pointing to the apache4 ant installation directory
• Below is the new build.xml file you will need.

<project name="go-ivy-with-ivy.xml" default="go" xmlns:ivy="antlib:org.apache.ivy.ant" basedir=".">
<!-- here is the version of ivy we will use. change this property to try a newer
version if you want -->
<property name="ivy.install.version" value="2.1.0" />
<property name="ivy.jar.dir" value="${basedir}/ivy" />
<property name="ivy.jar.file" value="${ivy.jar.dir}/ivy.jar" />

<property name = "ivy.lib.dir" value="${basedir}\newlib" />

<path id="lib.path.id">
       <fileset dir="${ivy.lib.dir}" />
</path>

<property name="build.dir" value="build" />
<property name="src.dir" value="src" />

<!-- =================================
 target: go
           Go ivy, go!
================================= -->
<target name="go" depends="resolve "
   description="--> resolve dependencies, compile and run the project">
<echo message="using ivy to resolve commons-lang 2.1 with ivy.xml"/>

<echo message="compiling..."/>
<mkdir dir="${build.dir}" />
<javac srcdir="${src.dir}" destdir="${build.dir}" classpathref="lib.path.id" />

<echo>
We are now ready to execute our simple program with its dependency on commons-lang.
Let's go!
</echo>
<java classname="example.Hello">
   <classpath>
       <path refid="lib.path.id" />
       <path location="${build.dir}" />
   </classpath>
</java>
</target>

<!-- =================================
 target: clean     
================================= -->
<target name="clean" description="--> clean the project">
<delete includeemptydirs="true" quiet="true">
   <fileset dir="${src.dir}" />
   <fileset dir="${build.dir}" />
</delete>
</target>

<target name="resolve" depends="" description="Resolve the dependencies">
<ivy:retrieve/>
</target>
</project>

And to the future. I have two immediate aims:

1. Get a build of a spring hibernate using Ivy
2. Get a project with dependant projects using spring and hibernate using ivy.

oracle - listing all tables in a schema

select table_name,

to_number( extractvalue( xmltype(

dbms_xmlgen.getxml('select count(*) c from '||table_name))

,'/ROWSET/ROW/C')) count

from user_tables;

courtesy of Burleson

SQL info for oracle/sqlserver/mysql

While performing some testing to ensure my app was compliant with multiple db flavours I came across this site.

It is a useful reference to quickly see how the different flavours perform some common operations.

http://www.sqlinfo.net/

humax PVRs and the digital switch over

I have two Humax PVR freeview recorders that have been running quite happily for several years.

The 4th November was our official DSO date for the north west and suddenly most of my BBC channels had gone.

My two PVRs are a new 9300T and an aging 8000T.

A retune on the 9300 brought BBC back but with BBC1 on channel 800. The 8000T could not find BBC anywhere.

The fix for the 9300 was to go to the Installation Menu, select the 'Default Setting' menu, enter the pin code and reset the box. Low and behold BBC1 was back on channel 1.

However the 8000 was proving more awkward. For this box I performed a software update request from tyhe system menu. Despite not saying that any of the versions had changed this fixed the problem and BBC1 is now back on channel 1.

In parrallel I did email humax on UKSupport@humax-digital.co.uk and congratulations to them I did get a very prompt reply but I had fixed the problem by the time I had read the email. For interest they said to do the following:

Please follow the steps below to perform a Default reset;

1. Power OFF the receiver

2. Disconnect the Aerial cable

3. Power On the receiver

4. Press MENU

5. Select Installation

6. Enter your password (default = 0000)

7. Select Default Setting

8. Select YES

9. Enter your password (Default = 0000)

10. When the receiver restarts, power off

11. Connect the Aerial Cable

12. Power ON - the receiver will then search for the channels.