Possible Oracle Service Bus/Weblogic Server SSL bug

We have encountered 'Connection reset' exceptions in production environments when the Oracle Service Bus sends web requests to IIS/WCF-based web services over an SSL connection. By capturing packets and decrypting the SSL stream we were able to establish the following chain of events:

• The OSB opens an SSL connection with the IIS/WCF web service
• The OSB sends many HTTP/SOAP requests over this connection
• Eventually the OSB sends a HTTP/SOAP request for which the IIS/WCF web service fails to reply - it appears to be a bug in IIS/WCF (or the application) that occurs after n number of requests are sent over the same SSL connection
• IIS times out the TCP connection by sending a TCP RST packet to the OSB
• The OSB re-establishes an SSL connection to the IIS/WCF web service, it then proceeds to send the HTTP header to the web service but fails to send the HTTP payload. Since the IIS/WCF webservice is expecting more data everything halts waiting for more data. After some time IIS closes the connection by sending a TCP RST packet
• The OSB continues to re-establish the SSL connection and only send the HTTP header, this occurs 5 times in total
• On the last TCP SSL connection tear down the OSB throws a 'Connection reset' exception
• Now the OSB re-establishes a SSL connection and continues to send the other requests in the queue - this works as expected until we reach n number of requests for the same SSL connection where the same thing happens again

The following exception is encountered after 5 SSL connection tear downs.

java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at weblogic.utils.io.ChunkedInputStream.read(ChunkedInputStream.java:159)
at java.io.InputStream.read(InputStream.java:85)
at com.certicom.tls.record.ReadHandler.readFragment(Unknown Source)
at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
at com.certicom.tls.record.ReadHandler.read(Unknown Source)
at com.certicom.io.InputSSLIOStreamWrapper.read(Unknown Source)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
at weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
at weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
at weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:431)
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:359)
at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:947)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.getResponse(HttpOutboundMessageContext.java:574)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.access$000(HttpOutboundMessageContext.java:76)
at com.bea.wli.sb.transports.http.HttpOutboundMessageContext$RetrieveHttpResponseWork.handleResponse(HttpOutboundMessageContext.ja\
va:898)
at weblogic.net.http.AsyncResponseHandler$MuxableSocketHTTPAsyncResponse$RunnableCallback.run(AsyncResponseHandler.java:531)
at weblogic.work.ContextWrap.run(ContextWrap.java:41)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173

Our workaround was to set a timeout of 30 seconds in the OSB business service, and a retry count of 1. Since the IIS time out was approximately two minutes this meant that when it failed to respond, the OSB would tear down the connection and try again once more. This meant we never arrived in the state where the OSB was sending the header alone.

Groovy Logging Injection - AST Transformation

Following on from the theme of AST transformations, I created a global AST transform that would iterate through all classes and effectively inject the following field:

//SLF4J Logger/LoggerFactory
Logger logger = LoggerFactory.getLogger(this.class) 

Like with Grails, this means that a class can simply call 'logger' without worrying about creating it (or it's implementation for that matter). The following classes achieve this:

package annotations

import org.codehaus.groovy.transform.ASTTransformation
import org.codehaus.groovy.control.SourceUnit
import org.codehaus.groovy.ast.ASTNode
import org.codehaus.groovy.transform.GroovyASTTransformation
import org.codehaus.groovy.control.CompilePhase
import org.codehaus.groovy.ast.ClassNode
import org.codehaus.groovy.ast.FieldNode
import org.codehaus.groovy.ast.expr.MethodCallExpression
import org.codehaus.groovy.ast.expr.Expression
import org.codehaus.groovy.ast.expr.ClassExpression
import org.codehaus.groovy.ast.expr.ArgumentListExpression
import org.codehaus.groovy.ast.expr.ConstantExpression
import org.codehaus.groovy.ast.expr.PropertyExpression
import org.codehaus.groovy.ast.expr.VariableExpression
import java.lang.annotation.ElementType
import java.lang.annotation.Target
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Retention

//Injects the following into each non-static, non-enum class
// Logger logger = LoggerFactory.getLogger(this.class)

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
class LoggerInjectionASTTransformation implements ASTTransformation {
    public void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {

        List classes = sourceUnit.getAST()?.getClasses()
        classes?.each { ClassNode classNode ->
            if (classNode.isEnum() || classNode.isStaticClass() || classNode.isInterface() || classNode.isAnnotationDefinition()) return //skip to next class

            //ignore these transformations!
            if (classNode.declaresInterface(new ClassNode(org.codehaus.groovy.transform.ASTTransformation.class))) return

            //don't add another logger field (allows for overriding)
            if (classNode.getField("logger")) return

            //exclude classes decorated with the ExcludeLoggerInjection class
            if (classNode.getAnnotations(new ClassNode(ExcludeLoggerInjection.class))) return

            Expression objectExpression = new ClassExpression(new ClassNode(org.slf4j.LoggerFactory.class))
            PropertyExpression classReference = new PropertyExpression(new VariableExpression("this"),new ConstantExpression("class"))
            ArgumentListExpression arguments = new ArgumentListExpression(classReference)
            Expression initialValueExpression = new MethodCallExpression(objectExpression,"getLogger", arguments)

            FieldNode loggingField = new FieldNode("logger",2,new ClassNode(org.slf4j.Logger.class),classNode,initialValueExpression)

            classNode.addField(loggingField)

        }
    }
}

@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.TYPE])
public @interface ExcludeLoggerInjection {
}

Note that it only injects the logger field when there isn't already such a field (allowing the developer to override the logger implementation at compile-time if they desire), and it only adds a logger field to 'standard' classes, excluding the likes of interfaces and enums. Furthermore, an annotation 'ExcludeLoggerInjection' was created that, when decorated on a class, would stop the AST transform from injecting the logging class: this is especially useful when Java reflection is used to instantiate a class. One final step is required for invoking the global transform during class compilation. The classpath needs to contain a META-INF/services directory containing a file called 'org.codehaus.groovy.transform.ASTTransformation'. The contents of the file should contain all of the global annotation classed used during Groovy compilation. In our case the file simply contains:

nz.ac.auckland.mediaservices.digitizer.annotations.LoggerInjectionASTTransformation

If you're using Maven to do building this is pretty simple as resources are placed onto the classpath before compilation (we also need to build the annotation first before the remaining classes, see a previous post for how I used an additional ant groovyc task to force early compilation of these annotations).

Groovy AST Transform to inject Spring Security Check for Java Caller

This post follows on with the trend of using AST transforms to inject compile time behaviour into Groovy code so that Java code can exploit these benefits. In this case we have a piece of Java code (Spring RMIExporter) that calls methods on a 'service' class. Since it is straight Java making the call to the code we cannot use the Groovy metaclass approach of changing the way methods are called. The following AST transform was created that simply iterates over every public non-static class method and calls a security method provided as part of the annotation:

package annotations

import org.codehaus.groovy.transform.GroovyASTTransformationClass
import java.lang.annotation.ElementType
import java.lang.annotation.Target
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Retention
import org.codehaus.groovy.transform.GroovyASTTransformation
import org.codehaus.groovy.control.CompilePhase
import org.codehaus.groovy.ast.MethodNode
import org.codehaus.groovy.control.SourceUnit
import org.codehaus.groovy.ast.ASTNode
import org.codehaus.groovy.ast.builder.AstBuilder
import org.codehaus.groovy.transform.ASTTransformation

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
class SpringSecurityInvocationTransformation implements ASTTransformation {
    public void visit(ASTNode[] nodes, SourceUnit sourceUnit) {

        def instanceNode = nodes[1] //specified in documentation
        def annotationNode = nodes[0]

        String securityMethod = annotationNode.getMember("securityMethod").getValue()

        def interfaceMethods = instanceNode.getMethods()
        interfaceMethods.findAll { MethodNode methodNode ->
            methodNode.isPublic() && !methodNode.isStatic() && methodNode.declaringClass == instanceNode.getPlainNodeReference()
        }.each { MethodNode methodNode ->
            List currentStatements = methodNode.getCode().getStatements()
            String methodCallingCode = "this." + securityMethod + "()"

            def methodCallNodes = new AstBuilder().buildFromString(CompilePhase.SEMANTIC_ANALYSIS,true, methodCallingCode)
            currentStatements.add(0, methodCallNodes[0].statements[0])
        }
    }
}


@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.TYPE])
@GroovyASTTransformationClass(["annotations.SpringSecurityInvocationTransformation"])
public @interface SpringSecurityInvocation {
    String securityMethod();
}

It might have made more sense to do it over each method from the interface that the class implemented (seeing as it was an implemention of a RMI-exported interface). We can then invoke this using a class definition like:

import annotations.SpringSecurityInvocation

//this annotation injects the checkSecurity method at the start of
//every public non-static method for this class
@SpringSecurityInvocation(securityMethod="checkSecurity")
public class SampleRMIInterfaceImplementation implements SampleRMIInterfaceImplementation {
    private void checkSecurity() {
        //do security stuff
        //throw exception if invalid security
    }

    public void someRMIOperation() {
        //do stuff
    }
}

In this case someRMIOperation will first call the checkSecurity() method.

Groovy injecting JAX-WS annotations to interfaces for CXF web services

So having done some Spring development using CXF web services I found that interface method parameters were named arg0, arg1 etc in the WSDL. This is ugly but it seems that there isn't an elegant way of specifying parameter names using the basic CXF front end module (JAX-WS annotations do not work). So I switched to using JaxWsServerFactoryBean (with the security interceptors I discussed in a previous post). But this meant annotating the interface methods and parameters with JAX-WS annotations which was tedious and gross. So Groovy to the rescue with an AST transform that injects the annotations so that when the Java CXF code reads the interface it sees annotations. So the local AST transform iterates through every method in the interface and decorates the method with a javax.jws.WebMethod annotation containing the name of the method. It then iterates through the method's parameters and adds a javax.jws.WebParam annotation with the name of the parameter (this fixes the arg0, arg1 issue).

package annotations

import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Retention
import java.lang.annotation.ElementType
import java.lang.annotation.Target
import org.codehaus.groovy.transform.GroovyASTTransformationClass
import org.codehaus.groovy.transform.ASTTransformation
import org.codehaus.groovy.transform.GroovyASTTransformation
import org.codehaus.groovy.control.CompilePhase
import org.codehaus.groovy.control.SourceUnit
import org.codehaus.groovy.ast.ASTNode
import org.codehaus.groovy.ast.MethodNode
import org.codehaus.groovy.ast.ClassNode
import org.codehaus.groovy.ast.AnnotationNode
import org.codehaus.groovy.ast.expr.ConstantExpression

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
class WSAnnotationTransformation implements ASTTransformation {
    public void visit(ASTNode[] nodes, SourceUnit sourceUnit) {
        def interfaceNode = nodes[1] //specified in documentation
        def interfaceMethods = interfaceNode.getMethods()
        interfaceMethods.each { MethodNode methodNode ->

            //add an annotation for the method web operation
            ClassNode webMethod = new ClassNode(javax.jws.WebMethod)
            AnnotationNode methodAnnotation = new AnnotationNode(webMethod)
            methodAnnotation.addMember("operationName",new ConstantExpression(methodNode.getName()))
            methodNode.addAnnotation(methodAnnotation)

            //add the annotation for the parameter names
            methodNode.parameters.each { parameter ->
                ClassNode webParam = new ClassNode(javax.jws.WebParam)
                AnnotationNode parameterAnnotation = new AnnotationNode(webParam)
                parameterAnnotation.addMember("name",new ConstantExpression(parameter.getName()))
                parameter.addAnnotation(parameterAnnotation)
            }
        }
    }
}

@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.TYPE])
@GroovyASTTransformationClass(["annotations.WSAnnotationTransformation"])
public @interface ParameterAnnotationInjection {
}

We can then decorate the interface with the annotation: @ParameterAnnotationInjection e.g.

package webservices

import annotations.ParameterAnnotationInjection

@ParameterAnnotationInjection
public interface DigitizerService {
     void doSomething(String a, String b, String c);
     void somethingElse(String a, String b, String c);
}

Which returns a XSD schema like:

Also, the Eclipse-Groovy Maven plugin had trouble with the annotation and compilation order, so I added an ant task to force the compile of the annotation before calling the Eclipse-Groovy Maven plugin to do the heavy lifting:

            
                maven-antrun-plugin
                
                    
                        process-sources
                        process-sources
                        
                            
                                
                                
                                         
                                
                                
                                
                                    
                                
                            
                        
                        
                            run
                        
                    
                
            

Weblogic JMS message receiver

Following on from an earlier post where I wrote a quick piece of java to put files onto an arbitrary JMS topic/queue, I also wanted something to read text documents off a topic/queue. I wrote the following code that takes a topic/queue and continues to read messages off of it, it also supports durable topic subscription IDs. Remember to include wlclient.jar, wljmsclient.jar in the classpath when building/running this (todo: mavenize all of these Weblogic libraries).

import javax.jms.*;
import javax.naming.*;
import java.util.Properties;
import java.io.*;

public class JMSConsumeMessage {

    public static void main(String[] args) throws Exception {
		if (args.length < 5 || args.length > 6) {
			System.out.println ("Usage: TOPIC|QUEUE ProviderURL UserName Password DestName [subscriptionName]");
			return;
		}

		String destType = args[0];
		String provider = args[1];
		String username = args[2];
		String password = args[3];
		String destName = args[4];
		String subscriptionName = null;
		if (args.length == 6) subscriptionName = args[5];

		if (!(destType.equals("QUEUE") || destType.equals("TOPIC"))) throw new Exception ("Must specify destination as TOPIC or QUEUE");

		Properties props = System.getProperties();
		props.put(Context.INITIAL_CONTEXT_FACTORY,"weblogic.jndi.WLInitialContextFactory");
		props.put(Context.PROVIDER_URL, provider);
		props.put(Context.SECURITY_PRINCIPAL, username);
		props.put(Context.SECURITY_CREDENTIALS, password);

		InitialContext ctx = new InitialContext(props);

		Session session = null;
		MessageConsumer consumer = null;
		Connection connection = null;

		try {
			if (destType.equals("TOPIC")) {
				TopicConnectionFactory tconfactory = (TopicConnectionFactory)ctx.lookup("javax.jms.TopicConnectionFactory");
				TopicConnection tcon = tconfactory.createTopicConnection();
				if (subscriptionName != null) tcon.setClientID(subscriptionName);
				TopicSession tsession = tcon.createTopicSession(false, Session.AUTO_ACKNOWLEDGE);
				Topic topic = (Topic)ctx.lookup(destName);
				TopicSubscriber tsubscriber = (subscriptionName == null ? tsession.createSubscriber(topic) : tsession.createDurableSubscriber(topic,subscriptionName));
				session = (Session)tsession;
				consumer = (MessageConsumer)tsubscriber;
				connection = (Connection)tcon;
			} else if (destType.equals("QUEUE")) {
				QueueConnectionFactory qconfactory = (QueueConnectionFactory)ctx.lookup("javax.jms.QueueConnectionFactory");
				QueueConnection qcon = qconfactory.createQueueConnection();
				QueueSession qsession = qcon.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
				Queue queue = (Queue)ctx.lookup(destName);
				QueueReceiver qreceiver = qsession.createReceiver(queue);
				session = (Session)qsession;
				consumer = (MessageConsumer)qreceiver;
				connection = (Connection)qcon;
			}

			connection.start();

			int currentIndex = 0;

			while (true) {
				Message m = consumer.receive();
				if (m instanceof TextMessage) {
					TextMessage txt = (TextMessage)m;
					BufferedWriter writer = new BufferedWriter(new FileWriter(destName + "-" + currentIndex + ".txt"));
					writer.write(txt.getText());
					writer.close();
					System.out.println("Read message to: " + destName + "-" + currentIndex + ".txt");
					currentIndex++;
				}
			}


		} finally {
		    if (consumer != null) consumer.close();
		    if (session != null) session.close();
		    if (connection != null) connection.close();
		}

    }

}

CXF, WS-Security and Spring Security

I wanted to find a way to integrate CXF and Spring Security so that CXF would take the WS-Security details and populate the SecurityContextHolder; leaving the service implementation to carry out authentication/authorisation processes using Spring Security features. I found a post on the Spring forums: http://forum.springsource.org/showthread.php?64492-WS-Security-integration that overrode the WSS4JInInterceptor and populated the SecurityContext here, but I wasn't totally happy with this solution: in theory it was possible that a different thread could handle a different interceptor in the chain further down the line possibly causing elevated privileges for a thread/caller that doesn't have the permissions. Furthermore the thread didn't remove the SpringContext details after execution which might cause security problems on outgoing interceptors if care wasn't taken. Inspired by the RMI ContextPropagatingRemoteInvocation offered by Spring I changed the CXF method invoker to set the SecurityContext for the calling thread, execute the method and then remove the SecurityContext. I made modifications to the request chain to make this happen; while all my code is in Groovy I've tried to make it as close to Java as possible, I suspect translation should be a trivial exercise.

I override the WSS4JInInterceptor to get the WS-Security details from the message, these are put into a Spring UsernamePasswordAuthenticationToken then placed into the messages's exchange under the key "WS_TOKEN". The CXF Architecture defines an exchange as holding "references to the in, out and fault messages for the current message exchange"; I wanted to include the value in the actual Message object but only the exchange was available to the method invocation (described later) - exchange would have to do. The following is the code for SpringAuthnInterceptor.groovy:

package ws

import org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor
import org.apache.cxf.binding.soap.SoapMessage
import org.apache.cxf.interceptor.Fault
import org.apache.ws.security.handler.WSHandlerConstants
import org.apache.ws.security.handler.WSHandlerResult
import org.apache.ws.security.WSSecurityEngineResult
import org.apache.ws.security.WSConstants
import org.apache.ws.security.WSUsernameTokenPrincipal
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken
import org.apache.cxf.ws.security.SecurityConstants

public class SpringAuthnInterceptor extends WSS4JInInterceptor {

    public SpringAuthnInterceptor() {
        super()
    }

    public SpringAuthnInterceptor(Map properties) {
        super (properties)
    }

    //with help from: http://forum.springsource.org/showthread.php?64492-WS-Security-integration
    public void handleMessage(SoapMessage message) throws Fault {
        message.put(SecurityConstants.VALIDATE_TOKEN, Boolean.FALSE);
        super.handleMessage(message)
        message.getContextualProperty(WSHandlerConstants.RECV_RESULTS).each { WSHandlerResult result ->
            result.getResults().each { WSSecurityEngineResult securityResult ->
                if ((securityResult.get(WSSecurityEngineResult.TAG_ACTION) & WSConstants.UT) > 0) {
                    WSUsernameTokenPrincipal principal = (WSUsernameTokenPrincipal)securityResult.get(WSSecurityEngineResult.TAG_PRINCIPAL)
                    if (!principal.getPassword()) principal.setPassword("")
                    UsernamePasswordAuthenticationToken usernameToken = new UsernamePasswordAuthenticationToken(principal.getName(), principal.getPassword())
                    message.getExchange().put("WS_TOKEN", usernameToken)
                }
            }
        }
    }
}

Once the user details are populated we need to override BeanInvoker to change the way that the WS method is invoked to add, then remove the SecurityContext. The following is SpringSecurityWSInvoker.groovy which carries this out:

package ws

import org.apache.cxf.service.invoker.BeanInvoker
import org.apache.cxf.message.Exchange;
import java.lang.reflect.Method
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken
import org.springframework.security.core.context.SecurityContextHolder

class SpringSecurityWSInvoker extends BeanInvoker {
    public SpringSecurityWSInvoker(Object proxy) {
        super(proxy)
    }

    protected Object performInvocation(Exchange exchange, final Object serviceObject, Method m,
        Object[] paramArray) throws Exception {
        Object invocationResult = null
        try {
            UsernamePasswordAuthenticationToken token = (UsernamePasswordAuthenticationToken)exchange.get("WS_TOKEN")
            if (token) {
                SecurityContextHolder.getContext().setAuthentication(token)
            }
            invocationResult = super.performInvocation(exchange,serviceObject,m,paramArray)
        } finally {
            SecurityContextHolder.clearContext()
        }

        return invocationResult
    }
}

Now we can use Spring to set up the relevant web service with the additional interceptor and overridden method invoker. I've used the following with the Spring DSL (I'm sure it translates to Spring XML easily also):

wsServerFactory(org.apache.cxf.frontend.ServerFactoryBean) {
        serviceClass = wsInterface
        address = "${ws.protocol}://${ws.host}:${ws.port}/${ws.serviceName}"
        invoker = new SpringSecurityWSInvoker(wsInterfaceInstance)
        inInterceptors = [new SpringAuthnInterceptor(["action":"UsernameToken","passwordType":"PasswordText"])]
        serviceName = new QName(ws.namespace,ws.serviceName,"")
    }

Where wsInterface is the webservice interface code that forms the WSDL and wsInterfaceInstance is the instantce of the implementation of this interface. To start the server the following code could be used:

def serverFactory = appContext.getBean("wsServerFactory")
serverFactory.create()