At the end, the author provides a PowerShell example using the Windows Azure Service Management CmdLets

Add-PSSnapin AzureManagementToolsSnapIn            

Get-HostedService $serviceName -Certificate $cert -SubscriptionId $subId |
Get-Deployment -Slot Production |
Set-DeploymentConfiguration `
  {$_.RolesConfiguration[$roleName].InstancesCount+=1}
            

Sometime back I had to delve into the depths of how one could dynamically and programmatically load a javaagent at runtime – in other words how a javaagent could be attached to a running process. Since then I’ve been meaning to blog my findings. Finally, after quite some time, I’ve got around to it but here I take the opportunity to expand the scope of the article to a complete treatment of how to create a javaagent, load it statically at startup or dynamically at runtime and also how to perform instrumentation of a simple class. As always the code will reside in an eclipse maven project that will be made available for download.

Introduction

Java agents are self contained components through which application classes pass at the level of byte code instructions in the form of byte arrays. They were introduced in java5 along with the powerful java.lang.instrument package. Java agents can be loaded statically at startup or dynamically (programmatically) at runtime to attach to a running process in a fail-safe fashion.

Lifecycle of a javaagent

The most important thing to understand is the lifecycle of a javaagent and its behaviour in relation to the application itself. The lifecycle hook points are as follows.

• PreMain – The PreMain method is invoked first prior to running the main method. This is the hookpoint for loading the javaagent statically at startup.
• Main – The main method is always invoked after the PreMain invocation.
• AgentMain – The AgentMain method can be invoked at any time before or after the Main method. This hookpoint is for loading the javaagent dynamically at runtime and attaching to a running process.

Creation of a javaagent

In order to create a java agent you must first define your chosen hook points from the lifecycle above as below.

package name.dhruba.javaagent;

import java.lang.instrument.Instrumentation;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyJavaAgent {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgent.class);

    private static Instrumentation instrumentation;

    /**
     * JVM hook to statically load the javaagent at startup.
     *
     * After the Java Virtual Machine (JVM) has initialized, the premain method
     * will be called. Then the real application main method will be called.
     *
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void premain(String args, Instrumentation inst) throws Exception {
        logger.info("premain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * JVM hook to dynamically load javaagent at runtime.
     *
     * The agent class may have an agentmain method for use when the agent is
     * started after VM startup.
     *
     * @param args
     * @param inst
     * @throws Exception
     */
    public static void agentmain(String args, Instrumentation inst) throws Exception {
        logger.info("agentmain method invoked with args: {} and inst: {}", args, inst);
        instrumentation = inst;
        instrumentation.addTransformer(new MyClassFileTransformer());
    }

    /**
     * Programmatic hook to dynamically load javaagent at runtime.
     */
    public static void initialize() {
        if (instrumentation == null) {
            MyJavaAgentLoader.loadAgent();
        }
    }

}

Once you’ve defined your hook points you must make the JVM aware by putting them in a manifest file.

Main-Class: name.dhruba.javaagent.MyMainClass
Agent-Class: name.dhruba.javaagent.MyJavaAgent
Can-Redefine-Classes: true
Can-Retransform-Classes: true
Premain-Class: name.dhruba.javaagent.MyJavaAgent

In the example project I’m doing so using maven and the maven assembly plugin which also packages the project as a single all inclusive uber executable jar for ease of testing.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-assembly-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>attached</goal>
      </goals>
      <phase>package</phase>
      <configuration>
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
        <archive>
          <manifest>
            <mainClass>name.dhruba.javaagent.MyMainClass</mainClass>
          </manifest>
          <manifestEntries>
            <Premain-Class>name.dhruba.javaagent.MyJavaAgent</Premain-Class>
            <Agent-Class>name.dhruba.javaagent.MyJavaAgent</Agent-Class>
            <Can-Redefine-Classes>true</Can-Redefine-Classes>
            <Can-Retransform-Classes>true</Can-Retransform-Classes>
          </manifestEntries>
        </archive>
      </configuration>
    </execution>
  </executions>
</plugin>

Instrumenting classes

Within the javaagent hook points one can define java.lang.instrument.ClassFileTransformer implementations that are invoked for all classes being loaded within an application. These receive classes as byte arrays and have the option of redefining them also in terms of byte arrays. Here I provide an example class file transformation.

The following User pojo returns ‘foo’ as its name.

package name.dhruba.user;

public class MyUser {

    public String getName() {
        return "foo";
    }

}

The following class file transformer (using ASM) redefines the User pojo to return ‘bar’ instead.

package name.dhruba.javaagent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.security.ProtectionDomain;

import org.objectweb.asm.ClassWriter;
import org.objectweb.asm.Label;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyClassFileTransformer implements ClassFileTransformer, Opcodes {

    static final Logger logger = LoggerFactory.getLogger(MyClassFileTransformer.class);

    @Override
    public byte[] transform(ClassLoader loader, String className, Class classBeingRedefined,
            ProtectionDomain protectionDomain, byte[] classfileBuffer)
            throws IllegalClassFormatException {
        logger.info("class file transformer invoked for className: {}", className);

        if (className.equals("name/dhruba/user/MyUser")) {

            ClassWriter cw = new ClassWriter(0);
            MethodVisitor mv;

            cw.visit(V1_6, ACC_PUBLIC + ACC_SUPER, "name/dhruba/user/MyUser", null,
                    "java/lang/Object", null);

            cw.visitSource(null, null);

            {
                mv = cw.visitMethod(ACC_PUBLIC, "", "()V", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(3, l0);
                mv.visitVarInsn(ALOAD, 0);
                mv.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "", "()V");
                mv.visitInsn(RETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            {
                mv = cw.visitMethod(ACC_PUBLIC, "getName", "()Ljava/lang/String;", null, null);
                mv.visitCode();
                Label l0 = new Label();
                mv.visitLabel(l0);
                mv.visitLineNumber(6, l0);
                mv.visitLdcInsn("bar");
                mv.visitInsn(ARETURN);
                Label l1 = new Label();
                mv.visitLabel(l1);
                mv.visitLocalVariable("this", "Lname/dhruba/user/MyUser;", null, l0, l1, 0);
                mv.visitMaxs(1, 1);
                mv.visitEnd();
            }
            cw.visitEnd();

            return cw.toByteArray();
        }

        return classfileBuffer;
    }

}

Note that this is not standard AOP or proxy logic. The class file transformer is literally redefining the bytecode instructions incrementally in the form of byte arrays.

Static loading of a javaagent at startup

Static loading of a javaagent is done by using the -javaagent=/path/to/file.jar=options command line option to the java executable as below.

$ java -javaagent:target/javaagent-examples-jar-with-dependencies.jar=foobarbaz name.dhruba.javaagent.MyMainClass foo bar baz
21:37:50.783 [main] INFO  name.dhruba.javaagent.MyJavaAgent - premain method invoked with args: foobarbaz and inst: sun.instrument.InstrumentationImpl@1786e64
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/javaagent/MyMainClass
21:37:50.789 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: [foo, bar, baz]
21:37:50.789 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
21:37:50.800 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
21:37:50.801 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Above, the javaagent lifecycle is clearly visible. The premain (and not the agentmain) method is invoked first. The class file transformer is passed classes as they are loaded. The transformer chooses to redefine the User object prior to its use. Subsequently the classes loaded at shutdown also pass through the transformer.

Dynamic loading of a javaagent at runtime

Dynamic loading of a javaagent at runtime can be done quite easily in a programmatic fashion but requires the sun tools jar to be present on the classpath. Certain libraries like jmockit have avoided this by opting to absorb the relevant classes from the sun tools jar into its library under the same package names. In Maven the tools jar can be added to the classpath very easily.

<dependency>
  <groupId>com.sun</groupId>
  <artifactId>tools</artifactId>
  <version>1.6.0</version>
  <scope>system</scope>
  <systemPath>${java.home}/../lib/tools.jar</systemPath>
</dependency>

The sun tools jar api can then be used to load the java agent simply by provided the path to the jar file as follows.

package name.dhruba.javaagent;

import java.lang.management.ManagementFactory;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.sun.tools.attach.VirtualMachine;

public class MyJavaAgentLoader {

    static final Logger logger = LoggerFactory.getLogger(MyJavaAgentLoader.class);

    private static final String jarFilePath = "/home/dhruba/.m2/repository/"
            + "javaagent-examples/javaagent-examples/1.0/"
            + "javaagent-examples-1.0-jar-with-dependencies.jar";

    public static void loadAgent() {
        logger.info("dynamically loading javaagent");
        String nameOfRunningVM = ManagementFactory.getRuntimeMXBean().getName();
        int p = nameOfRunningVM.indexOf('@');
        String pid = nameOfRunningVM.substring(0, p);

        try {
            VirtualMachine vm = VirtualMachine.attach(pid);
            vm.loadAgent(jarFilePath, "");
            vm.detach();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

As you can see – the code above is querying the pid of the running process and attaching the javaagent to the process. If any kind of failure occurs at this point it is ignored silently whereas if an agent is loaded on startup failures result in termination of startup.

We can now initialise the java agent prior to invoking our main method using the MyJavaAgent.initialize() hookpoint we declared earlier.

package name.dhruba.javaagent;

import java.util.Arrays;

import name.dhruba.user.MyUser;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyMainClass {

    static final Logger logger = LoggerFactory.getLogger(MyMainClass.class);

    static {
        MyJavaAgent.initialize();
    }

    /**
     * Main method.
     *
     * @param args
     */
    public static void main(String[] args) {
        logger.info("main method invoked with args: {}", Arrays.asList(args));
        logger.info("userName: {}", new MyUser().getName());
    }

}

The output is very similar but with a subtely different path through the code.

20:58:50.923 [main] INFO  n.dhruba.javaagent.MyJavaAgentLoader - dynamically loading javaagent
20:58:51.249 [Attach Listener] INFO  name.dhruba.javaagent.MyJavaAgent - agentmain method invoked with args:  and inst: sun.instrument.InstrumentationImpl@c2ff5
20:58:51.266 [main] INFO  name.dhruba.javaagent.MyMainClass - main method invoked with args: []
20:58:51.267 [main] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: name/dhruba/user/MyUser
20:58:51.276 [main] INFO  name.dhruba.javaagent.MyMainClass - userName: bar
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown
20:58:51.276 [DestroyJavaVM] INFO  n.d.javaagent.MyClassFileTransformer - class file transformer invoked for className: java/lang/Shutdown$Lock

Note that this time the agentmain method was invoked instead of the premain method. The rest of the output is the same.

Conclusion

Javaagents along with the java.lang.instrument package are a powerful feature set of the Java language that allow you complete control over the classes of any given application. Instrumentation also has far more liberty with its capabilities than proxying or pointcut weaving. This gives complete dynamic capability to an otherwise very static language. It has allowed development of powerful non-intrusive tools like JMockit which immediately gains numerous advantages over other mocking tools largely because it is based on instrumentation. I look forward to further exploring the possibilities going forward.

Resources

ASM, ByteCodeOutline Plugin for Eclipse (Eclipse 3.5 update site), java.lang.instrument, JMockit.

Over the past few months we’ve faced significant hurdles integrating jmockit into our development environment at a client-site primarily but not entirely because it is not a conventional development environment. The most noteable of these has been the inability to load the jmockit jar file from a mounted drive which is a remote file system. Having remote paths in the classpath such as //foo/bar/baz/jmockit.jar would result in a java.net.URI being constructed with that remote path and the URI being passed to the following java.io.File constructor.

public File(URI uri) {
    if (!uri.isAbsolute())
        throw new IllegalArgumentException("URI is not absolute");
    if (uri.isOpaque())
        throw new IllegalArgumentException("URI is not hierarchical");
    String scheme = uri.getScheme();
    if ((scheme == null) || !scheme.equalsIgnoreCase("file"))
        throw new IllegalArgumentException("URI scheme is not \"file\"");
    if (uri.getAuthority() != null)
        throw new IllegalArgumentException("URI has an authority component");
    if (uri.getFragment() != null)
        throw new IllegalArgumentException("URI has a fragment component");
    if (uri.getQuery() != null)
        throw new IllegalArgumentException("URI has a query component");
    String p = uri.getPath();
    if (p.equals(""))
        throw new IllegalArgumentException("URI path component is empty");
    p = fs.fromURIPath(p);
    if (File.separatorChar != '/')
        p = p.replace('/', File.separatorChar);
    this.path = fs.normalize(p);
    this.prefixLength = fs.prefixLength(this.path);
}

The following snippet in the above code would then result in an IllegalArgumentException being thrown with the message URI has an authority component and of course the authority component would be //foo.

if (uri.getAuthority() != null)
    throw new IllegalArgumentException("URI has an authority component");

This morning, however, I happened to notice the following extract in the jmockit changelog for the latest preview version – 0.997 although I’m not sure how long ago this was done.

A few fixes and enhancements to JMockit’s internal auto-initialization mechanisms: jmockit.jar (or a versioned jar name) can now be loaded from a remote file system without triggering a security-related exception

I wonder if this was in response to Shankar’s post on the obstacles we faced. It remains to be seen whether this indeed refers to the above issue and whether it resolves it. I’ll try it out once this version is finally released. Just for the record, to place renewed emphasis on the points made in Shankar’s article, the changes outlined in the following changelog entry under version 0.993 have caused us integration difficulties also.

Enhancement: to make JMockit easier to use when running under JDK 1.6+ without the -javaagent JVM parameter, two simplifications were introduced:

• The /lib/tools.jar file no longer needs to be added to classpath.
• Test classes no longer need to be annotated with @RunWith(JMockit.class) (for JUnit 4 tests), nor extend a JUnit/TestNG integration base class (JMockitTest, JMockitTestCase, JMockitTestNG).

There is a requirement for the second item when using JUnit, though: jmockit.jar needs to come before junit-4.x.jar in the classpath.

The above changes placed a constraint on classpath ordering and, by default, loaded all tests using the jmockit junit runner rather than the original junit runner and if jmockit wasn’t properly initialised then all tests would fail as long as jmockit was on the classpath. It’s a tricky problem to approach for jmockit. Making life easier for certain environments makes it more difficult in others. I would say that jmockit should allow the user to define how they would like their agent loaded and also have jmockit enabled explicitly only and not behind the scenes. Once integrated, however, it is a fine mocking library with a unique feature set.

As much as we all "try to be friends", the successor to Java (and there clearly needs to be one) "battle" continues

via blue train software

Groovy++ adds to Groovy static typing with very little in terms of trade off (meta programming)

• Allowing mixing of static and dynamic code in the same application.
• Same speed as Java – sometimes faster, sometimes slower depending on how the problem is expressed.
• Groovy++ is really Java ++ and is thus a natural and easy path for the millions of Java programmers.
• Groovy not being controlled by the boffins at Snoracle hopefully means innovation in the platform can happen much faster and address real developer needs.

Others in the community counter that Scala is ~50 times faster than Groovy++.

Seminar @Lab is Back

There’s going to be more seminars @Lab.  I’m looking forward to a 2 part series on high frequency trading and I’m gearing up to give some presentations myself.

Very Agile

This is not the place to ask for Agile or Scrum advice if you’re not serious about it.  A book inquiry got about 30 responses in no time flat.

Hosting

Lots of Labbers have advice on which hosting company to choose.  The most intriguing choice was Windows Azure.

The idea is to describe expectations from the object’s state prior to the method’s invocation, expectation from the input and expectation from state following the method’s invocation.
Since spring expression language exposes public methods, its interesting to see how we can verify correctness using other public methods.

For this purpose I will introduce 3 simple annotations:

@ConditionContext- a spring expression to be evaluated prior to any condition checks
@Precondition – a spring expression to be evaluated prior to the method invocation and after the latter
@Postcondition – a spring expression to be evaluated after the method invocation

All of these annotations are of the same code structure:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Postcondition {
String value();
}

I’ve written a very simple service that just manages a list of products.
The service supports the following operations:

public interface FooService {

public Set<Product> getProducts();

public Product getProduct(int id);

public void insertProduct(Product product);

public void updateProduct(Product product);

public void deleteProduct(Product product);

}

Where a Product just contains an ID and price.

Before I show the behind the scenes, I would like to show the final outcome which is what I described above:

@ConditionContext("#ctx.put('sizeBefore', #self.Products.size())")
@Precondition("#arg0 != null and #arg0.Id > 0")
@Postcondition("#sizeBefore + 1 == #self.Products.size() and " +
"#self.Products.?[Id == 1].size() == 1 and " +
"#self.getProduct(#arg0.Id).equals(#arg0)")
public voidinsertProduct(Product product) {
products.add(product);
}

Here are my expectations from the insertProduct method as described in the constraint (just as an example, i know there many other possible constraints which can be expressed):

As a precondition I want the input to be valid which means the product argument (arg0) is not and has a positive ID.
I can also express conditions about the service’s state prior to insert, I will show that in the post condition.
Following the insert the number of products should be incremented by 1, and I shall be able to find the added product.
The latter is expressed in two different ways:
1) the obvious one of invoking getProduct() on the Id and making sure the objects are equal.
2) getting the entire collection and finding it in the collection. I have also wanted to demonstrate SpEL, so I used the collection selection syntax it provides.
The condition context is simply a way to store information prior to method invocation.
Since you cant set variables outside the SpEL context, I used a map to store the data and I later introduce the map’s keys as variable in the SpEL context.
This is why in @Postcondition you see #sizeBefore.
How does this work?

I started by creating a small driver code:

ApplicationContext ctx = newClassPathXmlApplicationContext(newString[]{"beans.xml"});
FooService fooService = (FooService) ctx.getBean("fooService");
Product product1 = new Product();
product1.setId(1);
product1.setPrice(1);
fooService.insertProduct(product1);

Adding my AOP and bean definitions in beans.xml file:

<!-- Enabling aspect support -->
<aop:aspectj-autoproxy/>
<bean id="product" class="com.lab49.spelaspecttest.domain.Product"/>
<bean id="fooService" class="com.lab49.spelaspecttest.service.DefaultFooService"/>
<bean id="aspectBean" class="com.lab49.spelaspecttest.aspect.PointCuts"/>

I created an aspect with the following:

1. A Pointcut for any public method.
2. An Around advice (such that I can control method invocation) to handle the pre and post condition logic.
   The logic is quite simple, evaluate the preconditions, if they are met execute the method, if not throw an IllegalStateException.
   Following the method execution, evaluate the post conditions and do the same.

The real magic occurs in the aspect code as follows (see documentation in code):

@Aspect
public class PointCuts {

	@Pointcut("execution(public * *(..))")
	private void anyPublicOperation() {}

	@Pointcut("within(com.lab49.spelaspecttest..*)")
	private void inServiceLayer() { }

	/**
	 * Just for debugging, print methods executed in service layer
	 * @param jp
	 */
	@Before("inServiceLayer()")
	public void logServiceMethodExecution(JoinPoint jp) {
		System.out.println("Executing: " + jp.getSignature().getName());
	}	

	/**
	 * I use the public pointcut and add target and annotation to statically type the annotation variables and targen object (the service).
	 *
	 */
	@Around(value = "anyPublicOperation() && target(bean) && @annotation(conditionContext) && @annotation(precondition) && @annotation(postcondition)", argNames="bean, conditionContext, precondition, postcondition")
	public Object checkConditionConstraints(ProceedingJoinPoint pjp, Object bean, ConditionContext conditionContext, Precondition precondition, Postcondition postcondition) throws Throwable {
		System.out.println("Condition context: " + conditionContext.value());
		System.out.println("Precondition: " + precondition.value());
		System.out.println("Postcondition: " + postcondition.value());

		// SPeL action begins
		EvaluationContext context = new StandardEvaluationContext(bean);
		// I add a #self variable because the implicit this SPeL evaluates properties
		// with no object against can be confusing or inconsistent.
		context.setVariable("self", bean);
		// Add all the #arg* variables to the context, the method inputs.
		addArgumentsToContext(pjp, context);

		ExpressionParser parser = new SpelExpressionParser();
		Expression preConditionExp = parser.parseExpression(precondition.value());

		// a little syntactic sugar - we turn the keys in the map in the condition context
		// to variables in evaluating the precondition and postcondition expressions
		updateConditionContext(conditionContext, parser, context);

		// SPeL uses the class<T> generics to ease on casting return values
		boolean preconditionValue = preConditionExp.getValue(context, Boolean.class);
		System.out.println("Precondition value: " + preconditionValue);

		if(!preconditionValue) {
			throw new IllegalStateException("Precondition " + precondition.value() + " failed");
		}

		// use the JoinPoint to actually execute the underlying method
		Object retval = pjp.proceed();

		Expression postConditionExp = parser.parseExpression(postcondition.value());
		boolean postConditionValue = postConditionExp.getValue(context, Boolean.class);
		System.out.println("Postcondition value: " + postConditionValue);

		if(!postConditionValue) {
			throw new IllegalStateException("Postcondition " + postcondition.value() + " failed");
		}

		//if the postcondition is met, return the method's original return value
		return retval;
	}

	private void updateConditionContext(ConditionContext conditionContext, ExpressionParser parser, EvaluationContext context) {
		Map<String,Object> map = new HashMap<String, Object>();
		context.setVariable("ctx", map);
		Expression contextExpression = parser.parseExpression(conditionContext.value());
		contextExpression.getValue(context);

		for(Entry<String, Object> entry : map.entrySet()) {
			context.setVariable(entry.getKey(), entry.getValue());
		}
	}

	private void addArgumentsToContext(ProceedingJoinPoint pjp, EvaluationContext context) {
		Object[] args = pjp.getArgs();
		for(int i = 0; i < args.length; i++) {
			context.setVariable("arg" + i, args[i]);
		}
	}
}

Also of interest, for those can get themselves to care about this most mundane (yet somehow fascinating) of topics, is the market at InTrade for future tax rates. Perhaps the most interesting finding here is that the prediction for Highest Marginal Single-Filer Fed Income Tax Rate to be greater than 38% dramatically rises from 2010 to 2011. This is the expiration of the Bush tax cuts, and the traders’ beliefs that these cuts will not be extended at their current levels.

Mint, on the other hand, provides this rather fascinating infographic about where the money comes from to fund our federal government. This breaks down the share of the tax burden by income level.

~alex

Interested in BDD? Then check out MSpec, SpecFlow and SpecUnit.net

Maybe this article explains Microsoft’s lack of product in the real-time/streaming space? Microsoft has been somewhat late (again) to the game with their StreamInsight offering – especially now with the Sybase RAP + Aleri deal.

This announcement means that I can finally use DryadLinq for financial applications I was getting a little bored of StreamInsight anyway

He demonstrates modeling, a DSL (domain specific language), code savings, reduced testing burden, and consistency of the user experience.

One word: Yowza.

The relevant bit from the article:

New research suggests that when a home’s value falls below 75 percent of the amount owed on the mortgage, the owner starts to think hard about walking away, even if he or she has the money to keep paying.

In a situation without precedent in the modern era, millions of Americans are in this bleak position …

… The number of Americans who owed more than their homes were worth was virtually nil when the real estate collapse began in mid-2006, but by the third quarter of 2009, an estimated 4.5 million homeowners had reached the critical threshold, with their home’s value dropping below 75 percent of the mortgage balance …

… With figures released last week showing that the real estate market was stalling again, their numbers are now projected to climb to a peak of 5.1 million by June — about 10 percent of all Americans with mortgages.

~alex

Moles – Isolation framework for .NET

• .NET Framework 4.0: Comparing LINQ and PLINQ performance
• F# for Parallel and Asynchronous Programming – PDC 2009
• Steve Jobs and Control
• Tracing and Caching for Entity Framework available on MSDN Code Gallery – hopefully we’ll see Velocity (AppFabric) integration with other Microsoft products soon
• NDepend v3 is now 100% integrated in Visual Studio – Must Have VS Plugin?
• Are you using the CCR? – One runtime I just haven’t managed spend enough time with
• CLR V4: Stuff That May Break Your Profiler + Profiler Attach Part 2: Ok, now what?
• Much Ado About Monads – Creating Extended Builders
• An Insurgency of Quality
• WPF Unit Testing
• Line Charts with Data Templates – missed this one when I last looked at the MSDN Magazine site

For some reason I still seem to be in StreamInsight world. Not sure if that is a good thing or a bad thing. My Strategy Pricer POC is complete as far as I’m concerned, but another POC is bugging me. The problem I’ve walked into in the last few days is around ConfigInfo. In the StreamInsight samples that are supplied by Microsoft, the ConfigInfo parameter passed to CepStream.Create is a constructed object that passes configuration information to the input/output adapter. Now I was expecting this object to simply be passed to the adapter. However it appears I either have a very silly bug or for some reason I am seeing new instanced of my ConfigInfo class constructed by StreamInsight and passed to the adapters when the ConfigInfo class is more than the basic string property struct as per the StreamInsight samples.

Confused? Very

The problem is all centred around the queries being complied, and the ConfigInfo object being serialised. Here the stack trace that invokes the constructor of my ConfigInfo class:

System.Runtime.Serialization.dll!System.Runtime.Serialization.ClassDataContract.ReadXmlValue(System.Runtime.Serialization.XmlReaderDelegator xmlReader = {System.Runtime.Serialization.XmlReaderDelegator}, System.Runtime.Serialization.XmlObjectSerializerReadContext context) + 0x30 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.XmlObjectSerializerReadContext.ReadDataContractValue(System.Runtime.Serialization.DataContract dataContract, System.Runtime.Serialization.XmlReaderDelegator reader) + 0x10 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator reader, string name, string ns, ref System.Runtime.Serialization.DataContract dataContract) + 0x81 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(System.Runtime.Serialization.XmlReaderDelegator xmlReader, System.Type declaredType, System.Runtime.Serialization.DataContract dataContract, string name, string ns) + 0x28 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.DataContractSerializer.InternalReadObject(System.Runtime.Serialization.XmlReaderDelegator xmlReader, bool verifyObjectName) + 0xa8 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(System.Runtime.Serialization.XmlReaderDelegator reader, bool verifyObjectName) + 0x63 bytes
System.Runtime.Serialization.dll!System.Runtime.Serialization.DataContractSerializer.ReadObject(System.Xml.XmlReader reader) + 0x2a bytes


Sidebar: Maybe I should hook StreamInsight to NewsScope

It comes as no surprise that CEP vendors are getting acquired – Sybase Expected To Announce Acquisition of Aleri. As I’ve blogged about before, I expect the same thing to happen in the streaming real-time web space. With Sybase expected announcement, it should only help in forcing Microsoft to bring StreamInsight 2.0 to the table sooner rather than later – I know 1.0 isn’t out yet but 2.0 is always more interesting than 1.0

Looks like Sybase is in the process of acquiring Aleri.

It’ll be interesting to see how Sybase manages Aleri and Coral8 products, in both development and deployment (licensing).

Having said all of that, I think the iPad is a real disappointment as a home computer and internet-enabled device. Apple should un-invent it and release something worthwhile a year from now. I predict there will be many who feel similarly disappointed by the iPad, but will still buy it with the hope that support for the iPad today will enable Apple to bring something to market that is not a disappointment. I wish them good luck. Some others will probably get it because, if nothing else, it’s a cool touchscreen-toy.

PIMCO “looked closely at the way the housing market actually functioned at the microlevel. Their research included participating in “ride alongs” with local real estate and mortgage agents. Their findings … spoke clearly to the excesses taking place.

In and of itself perhaps not too astounding, but it’s implications are a bit astonishing.

1. During the housing bubble many investors were not doing the leg work to understand the very thing they were investing in.

2. And, (and perhaps more important from my perspective) how important doing the basic, almost ethnographic, research was to informing how PIMCO invested.

~alex

After upgrading to latest version of JMockit, version 0.996.0, we had ALL our unit tests failing with the following spurious error

java.lang.NoClassDefFoundError: org.junit.runner.Runner
 at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
 at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:24)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.(JUnit4TestReference.java:29)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestClassReference.(JUnit4TestClassReference.java:25)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.createTest(JUnit4TestLoader.java:40)
 at org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader.loadTests(JUnit4TestLoader.java:30)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:452)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
 at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

While my colleague who had been advocating the use of the utility asked me for a way to change the order of libraries in the classpath, we smelt rot and started debugging under the hood. After quite a bit of digging around, we found that the actual error was caused by the failure to load the JMockit agent dynamically. The following code is the sequence of calls which JMockit makes to load the agent.

 public class Startup {

   // other code omitted
   public static void verifyInitialization()
   {
      if (instrumentation == null) {
         new AgentInitialization().initializeAccordingToJDKVersion();
      }
   }
}

public final class AgentInitialization
{
   public void initializeAccordingToJDKVersion()
   {
      String jarFilePath = discoverPathToJarFile();
      if (Startup.jdk6OrLater) {
         new JDK6AgentLoader(jarFilePath).loadAgent();
      }
      else if ("1.5".equals(Startup.javaSpecVersion)) {
         throw new IllegalStateException(
            "JMockit has not been initialized. Check that your Java 5 VM has been started " +
            "with the -javaagent:" + jarFilePath + " command line option.");
      }
      else {
         throw new IllegalStateException("JMockit requires a Java 5 VM or later.");
      }
   }
   private String discoverPathToJarFile()
   {
      CodeSource codeSource = AgentInitialization.class.getProtectionDomain().getCodeSource();
      if (codeSource == null) {
         return findPathToJarFileFromClasspath();
      }

      URI jarFileURI; // URI is needed to deal with spaces and non-ASCII characters
      try {
         jarFileURI = codeSource.getLocation().toURI();
      }
      catch (URISyntaxException e) {
         throw new RuntimeException(e);
      }

      return new File(jarFileURI).getPath();
   }

   private String findPathToJarFileFromClasspath()
   {
      String[] classPath = System.getProperty("java.class.path").split(File.pathSeparator);
      for (String cpEntry : classPath) {
         if (cpEntry.matches(".*jmockit[-.\\d]*.jar")) {
            return cpEntry;
         }
      }
      return null;
   }
}

While this approach is non-intrusive and works out of the box in most cases, in large enterprise organisations, you are often limited extensively and work in a confined environment. My current environment loads files from a network file system and the following lines from the discoverPathToJarFile() cause the class initialization error

 jarFileURI = codeSource.getLocation().toURI(); 

returns

 CodeSource
file://remoterepository/libraries/jmockit/0.996.0/install/common/lib/jmockit.jar 

and

 return new File(jarFileURI).getPath(); 

returns

 IllegalArgumentException: URI has an authority component 

We had been using JMockit for a while now since 0.991.0 version and the agent loading had always been a problem. In our original usage, we had to do two things. We had to create our own version of the JDK6AgentLoader as this class has package level scope and create our own JUnit4 runner which loaded the agent from a hard-coded location, as below

public final class MyJMockitRunner extends BlockJUnit4ClassRunner
{
   static
   {
      MyJDK6AgentLoader.loadAgent(hardCodedPathtoAgentJar);
   }

   public JMockit(Class testClass) throws InitializationError
   {
      super(testClass);
   }
}

Why the problem now?

What changed majorly between the two versions was, Shadowing the org.junit.runner.Runner class which in the modified version loaded the java agent before initialization of the class.

public abstract class Runner implements Describable
{
   static
   {
      Startup.initializeIfNeeded();
   }

   public abstract Description getDescription();

   public abstract void run(RunNotifier notifier);

   public int testCount()
   {
      return getDescription().testCount();
   }
}

So even though we are trying to load the agent in our custom Runner, the loading of the Runner class itself preceding the sequence and aggressively loads the agent in its own strategy.

While this is a possible approach, I would highly criticize the usage of class shadowing for the following reasons

1. Open source API generally provide certain basic guarantees, and is generally tested extensively by the commnuity in different enviroments to ensure they work in the way they are designed. By shadowing a public class, you override this guarantee and for most common developers, the fact that they are using a modified class is oblivious. This can lead to late night debugging sessions and nightmares when things go wrong.
2. In this particular case, while only the tests that used JMockit should have failed, thousands of test cases across multiple projects failed at once. As a programmer, I always believe to lazily load resources and only when they are needed. But there is a necessity for mocking or not, the agent is going to be loaded whenever a Unit test is run. This is fundamentally wrong.
3. Any workarounds that had been applied to the library fail miserably with this aggressive approach to force your library to behave in a specific way.
4. While I agree that the owner cannot foresee all the environments his library would be used, trying to perform gimmicks under the hood on an open library could have been avoided.

Solution

While I’ve criticized the usage of shadowing a class, unfortunately the only solution that worked for us was to shadow the JMockit class that loaded the agent as below.

public final class AgentInitialization
{
   public void initializeAccordingToJDKVersion()
   {
 MyJDK6AgentLoader.loadAgent(hardCodedPathtoAgentJar);
   }
}

Better approach

There are a couple of suggestions for the JMockit API to overcome the shortcomings described above

1. Please do remove the shadowed JUnit class and make the agent loading explicit. This is already the case when you use the JMockit.class as the runner.
2. While the auto detection of the agent is brilliant, there could be some hooking mechanism which allowed the developers to extend it with their own strategies.

Last but not least, there definitely was an obvious solution to the problem, which was to explicitly specify the agent path in the VM arguments. The issue we had in that was there were too many OS, at least a few IDE, a shared continuous integration server and many many ant build scripts that need to be updated. It would have been a far more involved effort to get all this modified and maintained than applying a code fix which worked “under the hood”

When using the new System.Threading.Tasks.Task class in a WPF application, it’s not obvious how to make Tasks work well with the WPF UI thread.  DependencyProperty’s in .Net must be accessed from the UI thread.  Normally you would use Dispatcher.Invoke() or Dispatcher.BeginInvoke() to make sure an operation occurs on the correct thread.  The correct way to do this with Tasks is with TaskScheduler.FromCurrentSyncrhonizationContext() as the task scheduler for a Task.  You specify the scheduler when either creating a task or as part of a ContinueWith call on an existing Task.

The other piece of the puzzle is error handling with tasks.  If an exception is thrown inside a task, then that task catches the exception internally, stores it in Task.Exception and sets Task.IsFaulted = true.  When you continue from one task to the next using Task.ContinueWith() the error context is passed along with it.  This allows you to handle any background exception on the foreground thread in a central place instead of putting a try/catch block in the background Task.  If you do not check for the Task.Exception before the last chained task completes, and an exception was thrown in any of the chained tasks, then the exception will throw inside of the finalizer of the last Task in the chain! This feature is only for debugging purposes, so you MUST check for exceptions inside your last Task or catch all exceptions inside each Task. 

The following code snippet shows how it all works together:

QueryDone = false;
string arguments = "";
Task.Factory.StartNew(() =>
        //do work on background task
        DoDatabaseQuery(arguments)
    )
    //schedule this task on the UI thread after the
    //background task completes
    .ContinueWith(
        (task) =>
        {
            //must check for task fault so that exceptions
            //are handled correctly.
            //If task.Exception is not checked, then any
            //exceptions from a background task will be
            // rethrown in the task's finalizer
            if (task.IsFaulted)
            {
                MessageBox.Show(task.Exception.ToString());
            }

            QueryDone = task.IsCompleted;
        },
            //use the current synchronization context
            //as the task scheduler so that
            //continuation occurs on UI thread
            TaskScheduler.FromCurrentSynchronizationContext());

The Algoithmic Trading Working Group uploaded the v1.1 Release Candidate (internal build v2.7.0) for FIXatdl recently. You’ll need to register in order to download the files.

So start coding up those algo UIs already!

Here’s an earlier post that highlights some of the breaking changes with previous builds.

One official in Tamil Nadu was so stunned to receive the note that he handed back all the bribes he had solicited for providing electricity to a village. Another stood up, offered tea to the old lady from whom he was trying to extort money and approved a loan so her granddaughter could go to college.

The full story at the economist.

~alex

Where Everybody Knows Your Name

“I saw Alvin and The Chipmunks this weekend” – and they all laughed.

Some UXers were interested in

this color picker and this musical toy.

IPad

Lots of Lab Buzz around the IPad announcement, though we already knew this. More information about Apple’s design process, direct from Apple here and here. Overhyped? Underwhelming? Overpriced? Beautiful? Must have? Oh Yea.