Tuesday 26 July 2011

XQuery - finding changes in the trees

Given a set of node names I wanted to compare the data/text of these nodes from two structurally identical documents. This follows on from a previous post where we needed to translate the character sets for a given node's data/text; by comparing the before and after trees we can log what was actually changed before it was sent onto the next system.

The following XQuery takes a sequence of node names (as before), as well as the original and new documents; by comparing the relevant element's data it returns a string containing old and new values where they differ. This string is mainly useful for logging purposes.

declare namespace tmp = "http://www.tempuri.co.nz/xquery/";

declare function tmp:stringChangeList($nodeNames as xs:string*,$oldOutput as element(),$newOutput as element()) as xs:string* {
 for $curNodeName in $nodeNames
 let $oldValues := $oldOutput//*[name() = $curNodeName]
 for $oldValuePosition in (1 to count($oldValues))
 let $oldValueNode := $oldValues[$oldValuePosition]
 let $newValueNodeList := $newOutput//*[name() = $curNodeName]
 let $newValueNode := $newValueNodeList[$oldValuePosition]
 where data($oldValueNode) != data($newValueNode)
 return concat("(",$curNodeName," - Old: ",data($oldValueNode),", New: ",data($newValueNode),")")
};

Wednesday 20 July 2011

XQuery translations to Oracle Service Bus

So things are never as straight forward as they should be when Oracle Service Bus gets involved. It isn't possible to include modules in OSB XQuery... It is also difficult to include arbitrary documents without referring to an absolute path (deployment nightmare!); this was needed to refer to the 'translation table' XML file.

So I created a XQuery script compatible with Oracle Service Bus that does the following:
- Takes a node
- Takes a list of node names (including namespaces) that we want to convert
- A translation table as before
=> We return the provided node with the nodes in the node name list translated into the new values.

declare namespace tmp = "http://www.tempuri.co.nz/xquery/";

declare function tmp:convertString($inputString as xs:string, $translationSet as element()*) as xs:string {
        if (count($translationSet) = 0) then (: base case :)
           $inputString
        else (: recursion :)
           let $fromValue := data($translationSet[1]/translate_from)
           let $toValue := data($translationSet[1]/translate_to)
           let $updatedStr := fn:replace($inputString,$fromValue,$toValue)
           return tmp:convertString($updatedStr,remove($translationSet,1))
};

declare function tmp:replaceAll($element as element(), $updateNodes as xs:string*, $translationSet as element()*) as element() {
        element { fn:node-name($element) }
                     { $element/@*, for $child in $element/node()
                                    return if ($child instance of element())
                                           then tmp:replaceAll($child, $updateNodes,$translationSet)
                                           else if ($child instance of text() and count(index-of($updateNodes,fn:name($element))))
                                                then tmp:convertString($child,$translationSet)
                                                else $child
                     }
};

declare variable $element as element() external;
declare variable $updateNodes as xs:string* external;
declare variable $translationSet as element() external;

tmp:replaceAll($element, $updateNodes, $translationSet//translation)

Then in Oracle Service Bus I created a request/response pipeline at the very start with a replace node in each stage. The XPath is simply '.' in the variable body, with the XQuery created above. The external variables above are as follows:
translationSet:

        
          A
          BB
        
        
          C
          DD
        



updateNodes:
"firstname","lastname"

element:
$body

This will take the $body node, and find all nodes called "firstname" and "lastname", replacing all "A"'s and "C"'s with "BB" and "DD" respectively. Obviously strange characters will be more useful ;).

Tuesday 19 July 2011

XQuery to translate characters

We needed a way to translate characters from one character set to another depending on an external web service. I made the following xquery module uoa.xq:
module namespace uoa = "http://www.tempurl.co.nz/xquery";

declare function uoa:convertString($inputString as xs:string, $translationSet as element()*) as xs:string {
        if (count($translationSet) = 0) then (: base case :)
           $inputString
 else (: recursion :)
           let $fromValue := data($translationSet[1]/translate_from)
           let $toValue := data($translationSet[1]/translate_to)
           let $updatedStr := fn:replace($inputString,$fromValue,$toValue)
           return uoa:convertString($updatedStr,remove($translationSet,1))
};

Declaring a translation set can be done with an XML file like translation.xml:

 
          A
          BB
        
 
          C
          DD
        


Then invoking this can be achieved with the following xquery:
import module namespace uoa="http://www.tempurl.co.nz/xquery" at "uoa.xq";

let $inputString := "AABBCCDD"
let $translationSet := doc("translation.xml")/translation_set/translation
let $outputString := uoa:convertString($inputString,$translationSet)

return

        {$inputString}
        {$outputString}


This takes the input string and recursively iterates through the translation set (note that xquery is very functional and immutable variables mean that loops won't work!).

Monday 18 July 2011

Weblogic Server JMS message tester

I needed a way to send test messages through the CLI to Weblogic Server JMS modules (on server to test things were going as expected). I hacked together the following java code:
import javax.jms.*;
import javax.naming.*;
import java.util.Properties;
import java.io.*;

public class JMSSendMessage {

    public static void main(String[] args) throws Exception {
 if (args.length != 6) {
     System.out.println ("Usage: TOPIC|QUEUE ProviderURL UserName Password DestName FileName");
     return;
 }

 String destType = args[0];
 String provider = args[1];
 String username = args[2];
 String password = args[3];
 String destName = args[4];
 String filePath = args[5]; 

 if (!(destType.equals("QUEUE") || destType.equals("TOPIC"))) 
            throw new Exception ("Must specify destination as TOPIC or QUEUE");

 Properties props = System.getProperties();
 props.put(Context.INITIAL_CONTEXT_FACTORY,"weblogic.jndi.WLInitialContextFactory");
 props.put(Context.PROVIDER_URL, provider);
 props.put(Context.SECURITY_PRINCIPAL, username);
 props.put(Context.SECURITY_CREDENTIALS, password);

 InitialContext ctx = new InitialContext(props);

 Session session = null;
 MessageProducer producer = null;
 Connection connection = null;

 if (destType.equals("TOPIC")) {
     TopicConnectionFactory tconfactory = 
            (TopicConnectionFactory)ctx.lookup("javax.jms.TopicConnectionFactory");
     TopicConnection tcon = tconfactory.createTopicConnection();
     TopicSession tsession = tcon.createTopicSession(false, Session.AUTO_ACKNOWLEDGE);
     Topic topic = (Topic)ctx.lookup(destName);
     TopicPublisher tpublisher = tsession.createPublisher(topic);
     session = (Session)tsession;
     producer = (MessageProducer)tpublisher;
     connection = (Connection)tcon;
 } else if (destType.equals("QUEUE")) {
     QueueConnectionFactory qconfactory = 
            (QueueConnectionFactory)ctx.lookup("javax.jms.QueueConnectionFactory");
     QueueConnection qcon = qconfactory.createQueueConnection();
     QueueSession qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
     Queue queue = (Queue)ctx.lookup(destName);
     QueueSender qsender = qsession.createSender(queue);
     session = (Session)qsession;
     producer = (MessageProducer)qsender;
     connection = (Connection)qcon;
 }

 TextMessage msg = session.createTextMessage();

 connection.start();
 byte[] buffer = new byte[(int)new File(filePath).length()];
 BufferedInputStream bis = new BufferedInputStream(new FileInputStream(filePath));
 bis.read(buffer);
 bis.close();

 String textOutput = new String(buffer);

 System.out.println("Sending the following to: " + destName + 
             " (" + destType + ") on provider: " + provider);
 System.out.println(textOutput);

 msg.setText(textOutput);
 if (producer instanceof TopicPublisher) ((TopicPublisher)producer).publish(msg);
 else if (producer instanceof QueueSender) ((QueueSender)producer).send(msg);
 
 producer.close();
 session.close();
 connection.close();

 System.out.println("File sent successfully");

    }

}

You need two WLS files to compile it: wlclient.jar, wljmsclient.jar which you can 'borrow' from the WLS install directory.

Sunday 17 July 2011

LDAP membership search

A quick query to check the members of a particular LDAP group (or groups with * operator):
ldapsearch -x -H "ldaps://ldap-server:636" -D "cn=user,ou=ou,ou=ou,o=a"\
-w password -b "ou=group_list,dc=dc,dc=dc,dc=dc,dc=dc" "(cn=group_we_want)" member

Script to transfer SSH Keys

A simple script to send RSA public key to a remote SSH server so we can avoid typing a password.
#!/bin/bash                                 
cat ~/.ssh/id_rsa.pub | ssh $1 "mkdir .ssh; cat - >> ~/.ssh/authorized_keys; \
chmod a-w ~/.ssh; chmod og-rw ~/.ssh/authorized_keys"

Thursday 14 July 2011

Jenkins Apache Setup

After installing Jenkins with the MacOSX installer I wanted to put apache in the front end (in addition to RhodeCode). I had to modify the start up daemon script to listen to a different port as I use 8080 for grails dev. The following is my /Library/LaunchDaemons/org.jenkins-ci.plist:




 EnvironmentVariables
 
  JENKINS_HOME
  /Users/Shared/Jenkins/Home
 
 GroupName
 daemon
 KeepAlive
 
 Label
 org.jenkins-ci
 ProgramArguments
 
  /usr/bin/java
  -Xmx512m
  -Duser.home=/Users/Shared/Jenkins/Home
  -Djava.awt.headless=true
  -jar
  /Applications/Jenkins/jenkins.war
  --httpPort=8989
  --httpListenAddress=127.0.0.1
  --prefix=/jenkins
  --ajp13Port=-1
 
 RunAtLoad
 
 UserName
 daemon


Note that AJP couldn't be used due to an issue. Also note setting -Duser.home, this seems to be a problem where changing users causes Jenkins to put the .m2 Maven directory in the root's home directory which fails due to lack of permissions.

I then added the following to my SSL virtual host:

  ProxyPass http://127.0.0.1:8989/jenkins
  ProxyPassReverse http://127.0.0.1:8989/jenkins

Don't forget to update http://localhost:8989/jenkins/configure/ "Jenkins URL".

Restart Jenkins and Apache:
launchctl unload -w /Library/LaunchDaemons/org.jenkins-ci.plist
launchctl load -w /Library/LaunchDaemons/org.jenkins-ci.plist
sudo apachectl restart

RhodeCode Apache Proxy

I wanted to set up a proxy for SSL in front of my RhodeCode (and jenkins) setup using Apache in MacOSX. The following was added in production.ini under [app:main]:
filter-with = proxy-prefix

The following was added at the end:
[filter:proxy-prefix]
use = egg:PasteDeploy#prefix                                                                                                               
prefix = /rhodecode

Under /private/etc/apache2/httpd.conf:

             DocumentRoot "/Library/WebServer/Documents"
             ServerName host
             SSLEngine on
             SSLCertificateFile host.crt
             SSLCertificateKeyFile host.key
             SSLCertificateChainFile host.ca-bundle

             
                ProxyPass http://127.0.0.1:5000
                ProxyPassReverse http://127.0.0.1:5000
                RequestHeader set X-Url-Scheme https
             



Restart Apache with:
sudo apachectl restart

RhodeCode MacOSX Daemon/launchd

I wanted to make RhodeCode run as a launchd daemon process on MacOSX. This isn't as straight forward as it should be.
I made the following file in /Library/LaunchDaemons/rhodecode.plist




  EnvironmentVariables
  
    PYTHON_EGG_CACHE
    /opt/mercurial/.python-eggs
    CELERYD_PID_DIR
    /var/run
  

 GroupName
 daemon
 KeepAlive
 
 Label
 org.rhodelist
 ProgramArguments
 
  /usr/local/bin/paster
  serve
  /opt/mercurial/production.ini
 
 RunAtLoad
 
 UserName
 daemon
 


Restarting with the following commands:
launchctl unload -w /Library/LaunchDaemons/rhodecode.plist
launchctl load -w /Library/LaunchDaemons/rhodecode.plist

I also made a terrible hack to part of RhodeCode to force it to work under launchd (I may write a proper patch at some point in the future).
Line 40 of /Library/Python/2.6/site-packages/RhodeCode-1.1.8-py.2.6.egg/rhodecode/lib/pidlock.py
self.pidfile = "/var/run/" + self.pidfile

Yeah it's gross, but I can improve it later.