Bitbucket pipelines for java artifacts and uploading to AWS s3

 I am new to bitbucket and for whatever reason trying to create a pipeline that builds a java artifact & publishing to an s3 bucket has proved a lot harder than it should.

There were various gotchas.

1. artifacts created in one step are not visible in the next
2. the aws-s3-deploy step insisted on uploading absolutely everything

Anyway, I took the weekend off & resolved my remaining issues in about 5 minutes. Isn't always the way. So here is the bitbucket-pipelines.yml file I have ended up with.

#  Template maven-build

#  This template allows you to test and build your Java project with Maven.
#  The workflow allows running tests, code checkstyle and security scans on the default branch.

# Prerequisites: pom.xml and appropriate project structure should exist in the repository.

image: maven:3.6.3

pipelines:
  default:
    - parallel:
      - step:
          name: Build and Test
          caches:
            - maven
          script:
            - mvn -B verify --file pom.xml package
          artifacts:
            - $BITBUCKET_CLONE_DIR/target/v*
          after-script:
            - mkdir artifacts_for_deploy
            - mv target/v*war artifacts_for_deploy
            - pipe: atlassian/aws-s3-deploy:1.1.0
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                AWS_DEFAULT_REGION: 'us-west-1'
                S3_BUCKET:  $AWS_IFS_S3_ARTIFACT_BUCKET
                LOCAL_PATH: 'artifacts_for_deploy'
                COMMAND: 'upload'



      - step:
          name: checkstyle
          script:
            - pipe: atlassian/checkstyle-report:0.2.0
      - step:
          name: Security Scan
          script:
            # Run a security scan for sensitive data.
            # See more security tools at https://bitbucket.org/product/features/pipelines/integrations?&category=security
            - pipe: atlassian/git-secrets-scan:0.4.3

JDBI v3 and BatchChunkSize annotation, Only inserting up to the size of the BatchChunkSize

 Well hasn't it been a long time.

We recently upgraded to JDBI v3 and as part of that came across an interesting problem. The issue was that with v3 the insert was only inserting up to the number of rows defined in the annotation. So if the annotation was set at 50 you would get a max of 50 rows inserted.

in v2 we had a method on an interface defined as :

@GetGeneratedKeys 

@SqlBatch("Insert into my_table (......)

@BatchChunkSize(50)

void insertAsBatch(@BindBean("r") List<Foo> foos);

NB: in v2 this returned void

So innocently & clearly incorrectly I changed this to return 'int' assuming that would be the number of rows returned. Note that stupidly we had no test for this.

Users then found that we were only able to insert 50 rows.

Writing a test I then found that the value returned was the ID of the first row inserted. The fix was very simple. Change the return to an int[] IE this:

@GetGeneratedKeys 

@SqlBatch("Insert into my_table (......)

@BatchChunkSize(50)

int[] insertAsBatch(@BindBean("r") List<Foo> foos);

Apache webserver on windows (with same webapp repeated multiple times)

This post covers Apache 2.4 - the latest at the time of writing - Nov 2016

This is something I have wanted to set up for a long time to stop our users going directly to tomcat URLs with port numbers ie http://localhost:7070/foo and instead go to http://localhost/foo

So after a bit of searching & a lot of trial & error I appear to have it working.

We wanted to achieve two things:

1. use the default port 80
2. allow users to have the same web app installed on different tomcats/ports on the same server.

gotcha #1 -If you have Skype running - this has a nasty habbit (on windows atleast) of grabbing port 80.

• In Skype - click on Tools->Options->Advanved->Connection
• In there untick 'Use port 80 and 443 for additional incoming connections'
• stop & restart skype

Some useful resources - there are many ourt there but these two links are very informative:

• http://www.microhowto.info/howto/configure_apache_as_a_reverse_proxy.html
• http://www.apachetutor.org/admin/reverseproxies

Below I will attempt to describe the installation but to jump straight to the host config because this was where I spent most of my time.

gotcha #2 - unix is case sensitive, windows is not, but as I guess Apache is originally built for unix we have an odd interaction. Therefore for windows the Apache names & tomcat names MUST match each other completely both in character & case.

To create an Apache host 'bar' that has the same name (& case) as the tomcat  contrext is easy.

<Location "/bar">
  ProxyPass http://localhost:7070/bar 
  ProxyPassReverse http://localhost:7070/bar
</Location>

 To create an Apache host with a different name 'foo' proved hard, due something to do with how we were dealing with cookies & the session id of the logged in user. Without the two extra lines the jsessionid appeared in the URL and following other links would fail.

<Location "/foo">
  ProxyPass http://localhost:7070/bar 
  ProxyPassReverse http://localhost:7070/bar
  ProxyPassReverseCookiePath "/bar" "/foo" 
  ProxyPassReverseCookieDomain "/bar" "/foo" 
</Location>

Not sure why but with tomcat8 I have needed to add two extra lines:

<Location "/foo">
  ProxyPass http://localhost:7070/bar 
  ProxyPassReverse http://localhost:7070/bar
  ProxyPassReverseCookiePath "/bar" "/foo" 
  ProxyPassReverseCookieDomain "/bar" "/foo"
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1

gotch #3 - Pre Requisite VC++: Windows Apache needs Visual C++  You need to download & install the vc_redist.xNN.exe where NN is 64 or 86 depending if you are 64 bit or 32 bit.
Apache Installation

• Firstly a note, tomcat does not need to change at all & you will still be able to go to the tomcat URL as before.
• Getting Apache for windows. Going apache.org then sent me to Apache Lounge to get a pre built version. This is the reccomended option.
• I have installed two addional DLLs  in to the bin directory
• iconv.dll
• isapi_redirect.dll
• Pretty sure isapi_redirect.dll is not needed for me, but it is there & as yet I have not tried it without.
• I have created an ENV variable 'APACHE_WEB_SERVER' so that I can provide apache as a zip & IT can then install it whereever they feell fit.But gotch #4 , the ENV variable MUST use forward slashes eg: c:/utilisoft/Apache24/
• Once set up the service is created using the command:
• httpd -k install
• And uninstalled using:
• httpd -k uninstall

Now here is my httpd.conf. Hopefully you can see this OK:

#
# This is the main APACHE_WEB_SERVER HTTP server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.APACHE_WEB_SERVER.org/docs/2.4/> for detailed information.
# In particular, see 
# <URL:http://httpd.APACHE_WEB_SERVER.org/docs/2.4/mod/directives.html>
# for a discussion of each configuration directive.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "logs/access_log"
# with ServerRoot set to "/usr/local/APACHE_WEB_SERVER2" will be interpreted by the
# server as "/usr/local/APACHE_WEB_SERVER2/logs/access_log", whereas "/logs/access_log" 
# will be interpreted as '/logs/access_log'.
#
# NOTE: Where filenames are specified, you must use forward slashes
# instead of backslashes (e.g., "c:/APACHE_WEB_SERVER" instead of "c:\APACHE_WEB_SERVER").
# If a drive letter is omitted, the drive on which httpd.exe is located
# will be used by default.  It is recommended that you always supply
# an explicit drive letter in absolute paths to avoid confusion.

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# Do not add a slash at the end of the directory path.  If you point
# ServerRoot at a non-local disk, be sure to specify a local disk on the
# Mutex directive, if file-based mutexes are used.  If you wish to share the
# same ServerRoot for multiple httpd daemons, you will need to change at
# least PidFile.
#
#ServerRoot "c:/utilisoft/installed/webServers/APACHE_WEB_SERVERHttpd/APACHE_WEB_SERVER24"
ServerRoot "${APACHE_WEB_SERVER}"

#
# Mutex: Allows you to set the mutex mechanism and mutex file directory
# for individual mutexes, or change the global defaults
#
# Uncomment and change the directory if mutexes are file-based and the default
# mutex file directory is not on a local disk or is not appropriate for some
# other reason.
#
# Mutex default:logs

#
# Listen: Allows you to bind APACHE_WEB_SERVER to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent APACHE_WEB_SERVER from glomming onto all bound IP addresses.
#
#Listen 12.34.56.78:80
Listen 80

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
LoadModule access_compat_module modules/mod_access_compat.so
LoadModule actions_module modules/mod_actions.so
LoadModule alias_module modules/mod_alias.so
LoadModule allowmethods_module modules/mod_allowmethods.so
LoadModule asis_module modules/mod_asis.so
LoadModule auth_basic_module modules/mod_auth_basic.so
#LoadModule auth_digest_module modules/mod_auth_digest.so
#LoadModule auth_form_module modules/mod_auth_form.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_core_module modules/mod_authn_core.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_file_module modules/mod_authn_file.so
#LoadModule authn_socache_module modules/mod_authn_socache.so
#LoadModule authnz_fcgi_module modules/mod_authnz_fcgi.so
#LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule authz_core_module modules/mod_authz_core.so
#LoadModule authz_dbd_module modules/mod_authz_dbd.so
#LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_host_module modules/mod_authz_host.so
#LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule autoindex_module modules/mod_autoindex.so
#LoadModule buffer_module modules/mod_buffer.so
#LoadModule cache_module modules/mod_cache.so
#LoadModule cache_disk_module modules/mod_cache_disk.so
#LoadModule cache_socache_module modules/mod_cache_socache.so
#LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule cgi_module modules/mod_cgi.so
#LoadModule charset_lite_module modules/mod_charset_lite.so
#LoadModule data_module modules/mod_data.so
#LoadModule dav_module modules/mod_dav.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
#LoadModule dav_lock_module modules/mod_dav_lock.so
#LoadModule dbd_module modules/mod_dbd.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule dir_module modules/mod_dir.so
#LoadModule dumpio_module modules/mod_dumpio.so
LoadModule env_module modules/mod_env.so
#LoadModule expires_module modules/mod_expires.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
#LoadModule file_cache_module modules/mod_file_cache.so
#LoadModule filter_module modules/mod_filter.so
#LoadModule http2_module modules/mod_http2.so
LoadModule headers_module modules/mod_headers.so
#LoadModule heartbeat_module modules/mod_heartbeat.so
#LoadModule heartmonitor_module modules/mod_heartmonitor.so
#LoadModule ident_module modules/mod_ident.so
#LoadModule imagemap_module modules/mod_imagemap.so
LoadModule include_module modules/mod_include.so
#LoadModule info_module modules/mod_info.so
LoadModule isapi_module modules/mod_isapi.so
#LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
#LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
#LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
#LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
#LoadModule ldap_module modules/mod_ldap.so
#LoadModule logio_module modules/mod_logio.so
LoadModule log_config_module modules/mod_log_config.so
#LoadModule log_debug_module modules/mod_log_debug.so
#LoadModule log_forensic_module modules/mod_log_forensic.so
#LoadModule lua_module modules/mod_lua.so
#LoadModule macro_module modules/mod_macro.so
LoadModule mime_module modules/mod_mime.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule proxy_module modules/mod_proxy.so
#LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
#LoadModule proxy_express_module modules/mod_proxy_express.so
#LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_html_module modules/mod_proxy_html.so
LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_http2_module modules/mod_proxy_http2.so
#LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
#LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
#LoadModule ratelimit_module modules/mod_ratelimit.so
#LoadModule reflector_module modules/mod_reflector.so
#LoadModule remoteip_module modules/mod_remoteip.so
#LoadModule request_module modules/mod_request.so
#LoadModule reqtimeout_module modules/mod_reqtimeout.so
LoadModule rewrite_module modules/mod_rewrite.so
#LoadModule sed_module modules/mod_sed.so
#LoadModule session_module modules/mod_session.so
#LoadModule session_cookie_module modules/mod_session_cookie.so
#LoadModule session_crypto_module modules/mod_session_crypto.so
#LoadModule session_dbd_module modules/mod_session_dbd.so
LoadModule setenvif_module modules/mod_setenvif.so
#LoadModule slotmem_plain_module modules/mod_slotmem_plain.so
#LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
#LoadModule socache_dbm_module modules/mod_socache_dbm.so
#LoadModule socache_memcache_module modules/mod_socache_memcache.so
#LoadModule socache_shmcb_module modules/mod_socache_shmcb.so
#LoadModule speling_module modules/mod_speling.so
#LoadModule ssl_module modules/mod_ssl.so
#LoadModule status_module modules/mod_status.so
#LoadModule substitute_module modules/mod_substitute.so
#LoadModule unique_id_module modules/mod_unique_id.so
#LoadModule userdir_module modules/mod_userdir.so
#LoadModule usertrack_module modules/mod_usertrack.so
#LoadModule version_module modules/mod_version.so
#LoadModule vhost_alias_module modules/mod_vhost_alias.so
#LoadModule watchdog_module modules/mod_watchdog.so
LoadModule xml2enc_module modules/mod_xml2enc.so

<IfModule unixd_module>
#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
# It is usually good practice to create a dedicated user and group for
# running httpd, as with most system services.
#
User daemon
Group daemon

</IfModule>

# 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin admin@example.com

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
#
#ServerName www.example.com:80
ServerName localhost:80

#
# Deny access to the entirety of your server's filesystem. You must
# explicitly permit access to web content directories in other 
# <Directory> blocks below.
#
<Directory />
    AllowOverride none
    Require all denied
</Directory>



#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "${APACHE_WEB_SERVER}/htdocs"
<Directory "${APACHE_WEB_SERVER}/htdocs" >
    #
    # Possible values for the Options directive are "None", "All",
    # or any combination of:
    #   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
    #
    # Note that "MultiViews" must be named *explicitly* --- "Options All"
    # doesn't give it to you.
    #
    # The Options directive is both complicated and important.  Please see
    # http://httpd.APACHE_WEB_SERVER.org/docs/2.4/mod/core.html#options
    # for more information.
    #
    Options Indexes FollowSymLinks

    #
    # AllowOverride controls what directives may be placed in .htaccess files.
    # It can be "All", "None", or any combination of the keywords:
    #   AllowOverride FileInfo AuthConfig Limit
    #
    #AllowOverride None
    AllowOverride All

    #
    # Controls who can get stuff from this server.
    #
    Require all granted
</Directory>

#
# DirectoryIndex: sets the file that APACHE_WEB_SERVER will serve if a directory
# is requested.
#
<IfModule dir_module>
    DirectoryIndex index.htm
</IfModule>

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ".ht*">
    Require all denied
</Files>

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog "logs/error.log"

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel debug

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog "logs/access.log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "logs/access.log" combined
</IfModule>

<IfModule alias_module>
    #
    # Redirect: Allows you to tell clients about documents that used to 
    # exist in your server's namespace, but do not anymore. The client 
    # will make a new request for the document at its new location.
    # Example:
    # Redirect permanent /foo http://www.example.com/bar

    #
    # Alias: Maps web paths into filesystem paths and is used to
    # access content that does not live under the DocumentRoot.
    # Example:
    # Alias /webpath /full/filesystem/path
    #
    # If you include a trailing / on /webpath then the server will
    # require it to be present in the URL.  You will also likely
    # need to provide a <Directory> section to allow access to
    # the filesystem path.

    #
    # ScriptAlias: This controls which directories contain server scripts. 
    # ScriptAliases are essentially the same as Aliases, except that
    # documents in the target directory are treated as applications and
    # run by the server when requested rather than as documents sent to the
    # client.  The same rules about trailing "/" apply to ScriptAlias
    # directives as to Alias.
    #
    ScriptAlias /cgi-bin/ "${APACHE_WEB_SERVER}/cgi-bin/"

</IfModule>

<IfModule cgid_module>
    #
    # ScriptSock: On threaded servers, designate the path to the UNIX
    # socket used to communicate with the CGI daemon of mod_cgid.
    #
    #Scriptsock cgisock
</IfModule>

#
# "c:/APACHE_WEB_SERVER24/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "${APACHE_WEB_SERVER}/cgi-bin/">
    AllowOverride None
    Options None
    Require all granted
</Directory>

<IfModule mime_module>
    #
    # TypesConfig points to the file containing the list of mappings from
    # filename extension to MIME-type.
    #
    TypesConfig conf/mime.types

    #
    # AddType allows you to add to or override the MIME configuration
    # file specified in TypesConfig for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    # For type maps (negotiated resources):
    #AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    #AddType text/html .shtml
    #AddOutputFilter INCLUDES .shtml
</IfModule>

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
#MIMEMagicFile conf/magic

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# MaxRanges: Maximum number of Ranges in a request before
# returning the entire resource, or one of the special
# values 'default', 'none' or 'unlimited'.
# Default setting is to accept 200 Ranges.
#MaxRanges unlimited

#
# EnableMMAP and EnableSendfile: On systems that support it, 
# memory-mapping or the sendfile syscall may be used to deliver
# files.  This usually improves server performance, but must
# be turned off when serving from networked-mounted 
# filesystems or if support for these functions is otherwise
# broken on your system.
# Defaults: EnableMMAP On, EnableSendfile Off
#
#EnableMMAP off
#EnableSendfile on

# Supplemental configuration
#
# The configuration files in the conf/extra/ directory can be 
# included to add extra features or to modify the default configuration of 
# the server, or you may simply copy their contents here and change as 
# necessary.

# Server-pool management (MPM specific)
#Include conf/extra/httpd-mpm.conf

# Multi-language error messages
#Include conf/extra/httpd-multilang-errordoc.conf

# Fancy directory listings
#Include conf/extra/httpd-autoindex.conf

# Language settings
#Include conf/extra/httpd-languages.conf

# User home directories
#Include conf/extra/httpd-userdir.conf

# Real-time info on requests and configuration
#Include conf/extra/httpd-info.conf

# Virtual hosts
#Include conf/extra/httpd-vhosts.conf

# Local access to the APACHE_WEB_SERVER HTTP Server Manual
#Include conf/extra/httpd-manual.conf

# Distributed authoring and versioning (WebDAV)
#Include conf/extra/httpd-dav.conf

# Various default settings
#Include conf/extra/httpd-default.conf

# Configure mod_proxy_html to understand HTML4/XHTML1
<IfModule proxy_html_module>
</IfModule>
Include conf/extra/proxy-html.conf

# Secure (SSL/TLS) connections
#Include conf/extra/httpd-ssl.conf
#
# Note: The following must must be present to support
#       starting without SSL on platforms with no /dev/random equivalent
#       but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>

AcceptFilter http none
AcceptFilter https none
ProxyRequests off
ProxyPreserveHost off
RewriteEngine On


#EXAMPLE CONFIG
# 'foo' is the name that will be publicly available via the webserver
# If you want to keep the same name change 'foo' to 'bar'
#<Location "/foo">
#  ProxyPass http://localhost:7070/bar 
#  ProxyPassReverse http://localhost:7070/bar
#  ProxyPassReverseCookiePath "/bar" "/foo" 
#  ProxyPassReverseCookieDomain "/bar" "/foo" 
#</Location>

<Location "/bar">
  ProxyPass http://localhost:7070/bar 
  ProxyPassReverse http://localhost:7070/bar
</Location>

<Location "/foo">
  ProxyPass http://localhost:8080/bar 
  ProxyPassReverse http://localhost:8080/bar
  ProxyPassReverseCookiePath "/bar" "/foo" 
  ProxyPassReverseCookieDomain "/bar" "/foo" 
</Location>

Two useful tips:

• running 'httpd.exe -S' will show the state of the config & list some issues but not start the WebServer
• running 'httpd.exe' from the command line will also show up some issues whilst running the WebServer 

gotcha #4 - list your hosts in order by longest name first if the have common names

• see 'Worker Sharing' in the Apache docs 

Oracle - copying tables from one schema to another

There may well be other methods but this works for me.

This example copies some tables ('MY_TABLE1','MY_TABLE2',',MY_TABLE3') from a schema  named 'SRC_SCHEMA' to a schema 'DEST_SCHEMA'

1) Create a parameter file (called PARS.txt) describing the tables you want to copy:
   a) using a LIKE

schemas=SRC_SCHEMA dumpfile=some_tables.dmp INCLUDE=TABLE:"LIKE 'MY_TA_%'"

   b) here is an example listing specific tables

schemas=SRC_SCHEMA dumpfile=some_tables.dmp INCLUDE=TABLE:"IN ('MY_TABLE_1','MY_TABLE_2','MY_TABLE_3')"

2) export these tables using expdp

expdp system/password PARFILE=PARS.txt

3) import these tables from the created dump file using impdp.

NB1: there are other options to the parameter TABLE_EXISTS_ACTION. This one will simply overwrite what is there.

NB2: the example shows the remapping of two TABLESPACES as well

impdp system/password TABLE_EXISTS_ACTION=REPLACE dumpfile=some_tables.dmp REMAP_SCHEMA=SRC_SCHEMA:DEST_SCHEMA  REMAP_TABLESPACE=SRC_TSPACE1:DEST_TSPACE1   REMAP_TABLESPACE=SRC_TSPACE2:DEST_TSPACE2

Transient and still getting the error "MappingException: Could not determine type"

So I have a POJO with a field annotated as @Transient as in

private Workbook workbook;
    @Transient
    public Workbook getWorkbook() {
        return workbook;
    }
    public void setWorkbook(Workbook workbook) {
        this.workbook = workbook;
    }

But I am still getting the error:

Caused by: org.hibernate.MappingException: Could not determine type for: org.apache.poi.ss.usermodel.Workbook,

Solution: Check your import for @Transient.

If you are using 'import java.beans.Transient;' then you have the wrong one.

It wants to be 'import javax.persistence.Transient'

It looks like this error depends on the type of object you are making transient. If the object is String then it does not complain. In my case I had a poi Workbook class.

subclipse - saving or reusing your SVN repositories

So you want to reuse your list of SVN repositories in another WORKSPACE or in another install of eclipse.

If you use Subversive it is very easy to backup your SVN repositiories:

    This is done by simply going to File -> Export -> Select from General -> Preferences.
    Then on the next step you can tick from a list the preferences you want to export.
    In this list, SVN preferences and SVN repositories should be included.

I have found no way of doing this in Subclipse, so here is a manual method.
NB though it is a total copy, there is no way to just copy a limited set of SVN repositories.

    Navigate to the dir YOUR_WORKSPACE/.metadata/.plugins/org.tigris.subversion.subclipse.core       In there there is a file '.svnProviderState'
    This is a binary file but just copy this to the new WORKSPACE in the equivelent directory & that is it.

(Thanks to a colleague at work - Philip Lau for this)

oracle - listing locks & removing them

While logged in as sysadmin, SQL to list any locks on a particular table:

SELECT SID,SERIAL# 
FROM V$SESSION 
WHERE SID IN (SELECT SESSION_ID 
FROM DBA_DML_LOCKS 
WHERE NAME = 'SOME_TABLE_NAME');

SQL to remove a specific lock, using the values from the above SQL:

ALTER SYSTEM KILL SESSION 'SID,SERIALl#';

Hibernate - annotation mappings and misleading SerializationException

Took me a few hours to get to the bottom of this.
I was getting the following Exception, which did not make sense as the deSerialization test passed OK.
I have a domain object that I had proved through a unit test that I could serialize & deserialize successfully.


My domain object contained the following object which if I removed then worked OK:


But tomcat & my integration test were both failing with the following Exception.


It turns out I had missed out an annotation, obvious really but I could not spot the error & the Exception was totally misleading. What had I missed? The '@manyToOne' annotation. Twit.

Eclipse & the error 'Specified VM install not found: type Standard VM, name'

Scenario:

• I had a project that was previously building OK
• At some point I upgraded the version of java
• Came to rebuild the project and the Ant build was failing with the error popup
• Specified VM install not found: type Standard VM, name
• & it was referring to an old JDK that did not exist on my PC & was not referred to by eclipse with the list of installed JREs
• I searched in the project for the JDK listed 'jdk1.6.0_26' 
• No luck - could not find anythine
• I deleted the project & reloaded it from SVN
• Still No luck - I got the same error

After a bit of googling, I found this site http://www.javaworkspace.com/VMNotFound.do

I went for option '2' of edting the launch file in the directory simply renaming the JDK after stopping eclipse.

• C:\projects\WORKSPACE\.metadata\.plugins\org.eclipse.debug.core\.launches

All sorted now thanks.

sqlserver 2012 installation - a few possible gotchas

I have just installed sqlserver 2012 on the same server as one using 2008.
All was good as for accessing through both Management Studios but I struggled to get the web applications to connect to the 2012 databases.

These are all probably standard SQLServer Gotchas but they all got me.

Gotchas:

1. In the config I created a user 'sa' but this was disabled by default. 
1. find the user, right click & go to properties
2. click on 'Status' & select Enabled
2. The database it self did not allow Sql Server authentication.
1. Right click on the database server instance & select Properties
2. click on Security
3. Select the second option 'SQL Server & Windows Authentication'
3. Port Configuration
1. The default port for SQL server is 1433 so I needed to set up a different port for 2012
2. Run up the 'SQL Server Configuration Manager'
3. Inder the tab 'SQL Server Network Configuration' there will be two entries, one for the 2008 service & one for the 2012 service
4. Select tge 2012 service
5. select the TCP/IP protocal name & Right click & select Properties
6. Select the 'IP Addresses' tab
7. On mine the 'TCP Port' field was blank for all entries. I changed everyone to be 1533

Job done I can now connect with my user 'sa' using a standard conmection string but the new port number of 1533

sqlServer Management Studio not remembering users & passwords

Since I had a new Windows build for my PC I have had the very annoying bug that SQLServer Management Studio would not remember the username & passwords for hosts.

The Workarounds tab here provides the solution, but is essence it is this:

Rename:
SQL Server Management Studio 2008 delete the file C:\Users\%username%\AppData\Roaming\Microsoft\Microsoft SQL Server\100\Tools\Shell\SqlStudio.bin

SQL Server Management Studio 2005 delete the file – same as above answer but the Vista path. C:\Users\%username%\AppData\Roaming \Microsoft\Microsoft SQL Server\90\Tools\Shell\mru.dat

Editing Excel XLSX files manually to change cell label names

We have an application that manipulates XLSX Excel files using the excellent Java Apache POI package

We have the need to deploy this application for lots of different customers & the labels used in the spreadsheets need to be customer specific. In our case the cell names are prefixed with the customer name say 'foo' & we need to alter them all to 'foo'

Now you could do this longhand in Excel but this is time comsuming & a major issue for us is that you are not actually renaming the cells in Excel, you are actually adding  an extra name to a cell.

So what to do.

1) Handily XLSX files are acxtually zip compressed files so firstly extract all the files using your favourite tool, for me I use 7zip 
2) run a command to search for all the files that contain the name you want to change

•     find . | xargs grep -l foo 2> /dev/null

This will probably list 4 files:

• ./docProps/app.xml
• ./xl/charts/chart1.xml
• ./xl/charts/chart2.xml
• ./xl/workbook.xml 

3)Edit these files using your favourite editor. For the changes we want to make something like 'vi' or 'vim' is perfect.
4) If you are just changing names then one command will do it

•     :1,$s/foo/bar/g

5)We also had a case where cells were duplicated for two customers 'foo' & 'bar' and we wanted to remove all the 'foo' tags. This was in the files 'app.xml' & 'workbook.xml' . Here care is needed. Excel creates these files as one long line with loads of XML tags.

6) So firstly you have to seperate the lines, For app.xml this is:

• :1,$s/<\/vt:lpstr><vt:lpstr>/<\/vt:lpstr>\r<vt:lpstr>/g
  

    For workbook.xml, this two commands are needed:

• :1,$s/<\/definedName><definedName name/<\/definedName>\r<definedName name/g
  

• :1,$s/<definedNames><definedName name/<definedNames>\r<definedName name/g
  

7) You then delete the lines containing the string 'foo', but firstly count how many lines contain the string 'foo', so

• :%s/foo//gn

Then delete the lines

• :g/foo/d

8) In the case of app.xml two extra edits are required. See the XML below. This is a snippet of app.xml. In this example note the two numbers '87' & '93'. These both need decreasing by the numbers of lines you have deleted.Note I have formatted the XML below to make for easier reading

9) You now need to repackage this up using something like 7zip, but note the archive format must be 'zip' and the compression method must be 'DEFLATE'

Now go & enjoy.

Enhancing Apache DBUtils when column names do not match getter & setter names

In a recent post on running stored procedures I made use of Apache's DBUtils package as a basic ORM.

I had a problem where certain fields in my POJO were not getting populated.
The reason being that DBUtils does not make use of the '@Column' annotation and therefore some extra code is needed.

In our case it occurred for every column that had an underscore in the name but not in the POJO getters & setters.

A snipped from my POJO could look like this.


Using the same Dao in the post I refferred to above the ResultSetHandler will now be created as follows:

Hibernate - accessing database data using views

I have recently been working on a database for which I had no control
of the structure and it certainly was not laid out as I would have desired.

Interestingly the application that we were replacing had created convenience Views to access the data.
These Views were often complex typically including JOINS across several tables.

So it may be obvious but I decided to map some of my POJOs to these views.

A few provisos:

• The view needs to be designed such that there is still a column(s) that can be mapped to an ID.
• The views are clearly critical to the application so just as in the creation of tables the creation of these views MUST be under version control

Stored Procedures or functions that return POJO objects using hibernate

I have an application that needs to run some stored procedures that want to return List but I struggled with how to get this working.

To quote from the hibernate docs this is an area that they need to work on. It says:

27.5. Database functions

Warning

This is an area in Hibernate in need of improvement. In terms of portability concerns, this function handling currently works pretty well from HQL; however, it is quite lacking in all other aspects.

This is my eventual solution.

First off a Dao class

Here is the interface that the Dao will implement

Then the generic Dao MetOfficeDao
Note the following:

1. The use of ResultSetHandler from Apache is a basic ORM
2. JDBC parameters start from '1'
3. getSession().connection(); is deprecated

Just for completeness - here is an example stored procedure for SQLServer
Note the line:
SET NOCOUNT ON
This is needed so that the procedure just returns the data and not the number of rows returned

eclipse extension bundle timeout errors

If you find that eclipse is failing to install bundles or extensions with timeout errors then try adding this line to your INI file.

    -Dsun.net.client.defaultReadTimeout=30000

I use the spring sourse STS variant of eclipse so my file is called STS.ini but yours might be ECLIPSE.ini

what architecture is my windows PC

run this command

wmic os get osarchitecture

Version control of a database schema & it's procedures

I am lost for words having had an on-going battle at my current work.

This blog sums up my thoughts perfectly.

http://www.codinghorror.com/blog/2008/02/get-your-database-under-version-control.html

Viewing very large text files on windows

Just found this little app.

It's awesome - opening a 2GB file instantly.

Large Test File Viewer

android - rooting & upgrading

Not quite sure why I wrote this on my sister blog, but I did so here is the link