(\_ -> Phil Varner) » java

Java is dead/alive

Phil — Wed, 10 Dec 2008 07:59:56 +0000

Funniest thing I read today:

"Autoboxing was a misguided effort to paper over Java’s early decision to have a segregated type system for primitives and objects. It was Java’s Plessy v. Ferguson decision that pretended primitives and objects were separate but equal; but the claim was no more true in Java than it was in American jurisprudence."

I do love a good Supreme Court ruling analogy.

From one of my favorites, Elliotte Rusty Harold, in Java is Dead! Long Live Python!

Proleptic apoplexy

Phil — Sat, 12 Jul 2008 08:17:23 +0000

I spent a few hours this week trying to figure out why some date manipulation methods I was writing weren't working. In my test case, I had two instances of GregorianCalendar which I was comparing, an original and one that had been round-tripped through some conversion methods, via oracle.jbo.doman.Timestamp. Using the equals method, they were returning false. I used the toString method to see if there was something obviously different, but the strings were exactly the same. I even when so far as to step though equals in the debugger, until it disappeared into a JDK implementation class for which the source was unavailable. Even stranger, I discovered using compareTo returned 0. I set about printing each property of the instances until I discovered problem: the GregorianChange property.

In the tests, I was using XMLGregorianCalendar to take an ISO 8601 date string (e.g., 2006-09-22T00:00:00.000-00:00) and create a GregorianCalendar. The intention of this was to allow users to create statements like "if Order.timestamp > DateLib.from("2006-09-22T00:00:00.000-00:00"), do something". Below is the reduction of what was happening in the test and application code:

import java.util.GregorianCalendar;
import javax.xml.datatype.DatatypeFactory;
 
public class CalTest {
    public static void main(String[] args) throws Exception {
 
        String dtstring = "2006-09-22T00:00:00.000-00:00";
        GregorianCalendar gcFromXmlgc =
            DatatypeFactory.newInstance()
                .newXMLGregorianCalendar(dtstring).toGregorianCalendar();
        GregorianCalendar gcFromCons = new GregorianCalendar();
 
        gcFromCons.setTimeZone(gcFromXmlgc.getTimeZone());
        gcFromCons.setTimeInMillis(gcFromXmlgc.getTimeInMillis());
 
        System.out.println("equals? " + gcFromXmlgc.equals(gcFromCons));
        System.out.println("compareTo? " + gcFromXmlgc.compareTo(gcFromCons));
        System.out.println("XML GC: " + gcFromXmlgc.getGregorianChange());
        System.out.println("new GC: " + gcFromCons.getGregorianChange());
 
    }
}

Running this produces the following output:

equals? false
compareTo? 0
XML GC: Sun Dec 02 08:47:04 PST 292269055
new GC: Thu Oct 04 16:00:00 PST 1582

equals is false, but compareTo is 0? That's curious. The last two lines show why, now explained.

The core of the problem is XMLGregorianCalendar is intended to represent the W3C XML Schema 1.0 date/time datatypes. The date/time type in XML Schema essentially implements ISO 8601, with a few minor exceptions. The important part here is that it represents a "proleptic Gregorian calendar".

The Gregorian calendar is the calendar used exclusively in the West and by most of the global business world. Most people in the West don't even know there's even a name for the calendar, they just know it as "the calendar". The Gregorian calendar was adopted in 1582 to correct for the gradual drift caused by the previous Julian calendar. This change set the nominal date back 10 days and added a more astronomically appropriate leap year rule. The "proleptic" Gregorian calendar is then the Gregorian rules applied to dates before the calendar was adopted, as if the calendar had always been in effect. Therefore, all Western dates after 1582 are according to the Gregorian calendar, but any date before this can be in either the Julian calendar or the proleptic Gregorian calendar, with one always being specified explicitly.

Knowing this, one would then assume that the class GregorianCalendar implemented a proleptic Gregorian calendar. However, belying it's name, it actually represents a hybrid Julian-Gregorian, with each type having a set of rules as to the order of different dates, and the ability to set a date which defines when the rules cutover. The well-known Joda-Time library names these correctly, with a proleptic Gregorian, a postleptic Julian, and a hybrid GregorianJulian. The JDK 1.5 documentation includes this description, but it's not really useful unless you know what the semantics behind it mean. GregorianCalendar.toString() doesn't print it out, so you have to know it exists and to explicitly examine it when debugging. For most code using dates, the cutover date doesn't matter, since most software isn't going to be used for dates in the distant past, and if it is, there's hopefully a domain expert on hand to specify this sort of thing.

A default instance of GregorianCalendar implements a non-proleptic calendar — dates before to 4 October 1582 are in Julian, dates after are Gregorian. However, XMLGregorianCalendar implements a proleptic Gregorian calendar as specified by ISO 8601, so it sets the change date to Long.MIN_VALUE. The Timestamp class uses millis since the epoch (1970), so it will always give us a representation of the same date, but not the same GregorianCalendar. In Sun JDK 1.5, the only difference in equals between Calendar and GregorianCalendar is the change date:

 public boolean equals(Object obj) {
        return obj instanceof GregorianCalendar &&
        super.equals(obj) &&
            gregorianCutover == ((GregorianCalendar)obj).gregorianCutover;
    }

GregorianCalendar inherits compareTo (implementing Comparable) from Calendar, which simply compares the millis since the epoch. GNU Classpath 0.18 has a different implementation, as it doesn't take into account the change date:

  public boolean equals(Object o) {
    if (! (o instanceof GregorianCalendar))
      return false;
 
    GregorianCalendar cal = (GregorianCalendar) o;
    return (cal.getTimeInMillis() == getTimeInMillis());
  }

In writing my tests, I have made a huge error– I was testing for object equality, when I should have been testing for value ordering. I shouldn't have cared if the objects were the same, what I really cared about was that there was no distinguishable ordering between them– that both objects represented the same value.

I didn't actually write the tests from scratch, I had actually translated them from other date tests written in Oracle Business Rules RL, which as semantics similar to Java, but not exactly. The relevant difference here is that you can use the relative operators on Comparable implementors, which then are converted to calls to compareTo at runtime. I made the mistake of regex-replacing a bunch of obj1 != obj2 to !obj1.equals(obj2), when I should have done obj1.compareTo(obj2) != 0. If these had been any of the relative operators, I would have used compareTo, but != took me to !equals. Big mistake.

I learned a lot from this distraction, not just about calendars, but more importantly about equals and compareTo. I recently started reading the new version of Effective Java, so the sections on comparison and equality have new and deeper meaning for me now.

TestNG, part 2

Phil — Mon, 11 Feb 2008 05:27:30 +0000

Since migrating from JUnit 3, TestNG has been wonderful. Groups are the killer feature of TestNG that really make it worth the migration cost. When wanting to test a single method, I no longer need to manually comment or uncomment method names in the suite() method, I can just add a new group and run it from the command-line (well, from Ant. see below). Annotation-based test methods are much nicer, and have a much lower risk of accidentally being left out of the suite.

Just a few caveats before I show the Ant/TestNG setup we're currently using.

Merely moving to a new framework exposed several unintended test dependencies, so tests then failed because they ran after other tests. With the suite method, they always ran in the same order, so these dependencies were never found. None of ours were important, but there could have been ones that masked bugs.
Only void methods with names starting with "test" are annotated with @Test. This may seem obvious, but we had a few tests written by a developer auxiliary to our main team who had written a few tests that weren't prefixed properly, but ran because they were in the suite method. The JUnitConverter class should probably try and parse the suite method to find problems like this (Maybe I'll to a patch for this).
Only void returning methods annotated with @Test are run. Having a test method return a value doesn't generally make sense (it didn't in this case, either), but it may be difficult to understand if you test isn't running even though it's annotated.

Okay, so now onto the good stuff– our Ant/TestNG configuration. In these examples, I've replaced the name of my actual project with "foobar", and the prefix "sdk" indicates that it's the SDK part of the project.

First, in our common.xml file that is imported by all of our individual build.xml files, I added these lines, to define the location of the jar, add it to the common classpath, define the Ant task, and define the location for the reports to go ($twork is set to a temporary directory for the build):




  ....
  
  ....

Then in the build.xml for the specific tests, I added these targets. To clean, I added a target to delete old results:

Then I added a couple of targest to either produce a single "suc" (success) or "dif" (failure) file based on the results of the run (these files are used by the continuous build system to report the results of running the tests on a new build).

UPDATE:See this post for an updated version of the following targets.

Then, we have the target that actually calls the testng task. This target is never called directly, only through helper targets. Notice that it adds two listeners: one that will give use intermediate results on the command-line as the tests are running, and one that will give us a summary report at the end. After running, it then calls the previously mentioned targets. One thing I missed at first was that the test class files must be included in both the classpath (so the JVM can find them) and the classfileset element, so that TestNG will know what classes to use for the tests.

The next thing was to set up a few helper targets to call the run-testng target. The first was "run-testcase", which would allow you to run a group from only a specific TestCase class, for a feel similar to JUnit. This is run with the command line 'ant run-testcase -Dgroups=srg -Dtestcase=BazTestCase'. Note that the group "broken" is excluded by default. If you actually want to run the broken group, you call it with 'ant run-testcase -Dgroups=broken -Dtestcase=BazTestCase -Dexcluded-groups=""' to populate the property exclude-groups so it's redefinition is ignored. Also, we add the most used command-line target, rung. This called with "ant rung -Dgroups=srg", or more commonly when I'm using it, "ant rung -Dgroups=phil". I can just add my name to the groups for a test case, and easily run only that one while debugging code or writing new tests. This alone was worth the migration to TestNG– it's liberating when writing tests.

This is the listener that reports the intermediate results from running each test method. The name of the test class has its front chopped off so most of them will fit in an 80 character column, and it also prints the count of the tests (to gauge how far progressed the tests are) and the run-time for each test (to help gauge if there are any high-runtime/low-value tests out there). The one thing I would like to add but haven't looked at yet is printing out the actual results of the assert failure rather than just the stack trace of where it occured. I currently just look in the HTML report at the end for this.


package foobar.test.sdk;

import org.testng.*;

public class TestListener extends TestListenerAdapter {
    private int m_count = 0;

    private String name(ITestResult tr){
        return tr.getTestClass().getName().replaceAll("foobar\\.test\\
.","") +
            "." + tr.getMethod().getMethodName();
    }

    @Override
    public void onTestFailure(ITestResult tr) {
        log("[FAILED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSkipped(ITestResult tr) {
        log("[SKIPPED " + (m_count++) + "] => " + name(tr) );
    }

    @Override
    public void onTestSuccess(ITestResult tr) {
        log("[" + (m_count++) + "] => "+ name(tr) + " " + (tr.getEndMillis()-t
r.getStartMillis()) + "ms");
    }

    private void log(String string) {
        System.out.println(string);
    }
}

This is the reporter that I use for the summary report at the end of running all the tests:

package foobar.test.sdk;

import org.testng.*;
import java.util.*;

import static java.util.Arrays.asList;

public class SDKReporter implements IReporter {

    private String name(ITestResult tr){
        return tr.getTestClass().getName() + "." + tr.getMethod().getMethodNam
e();
    }

    public void generateReport(List xmlsuites ,List suites,String c) {

        for (ISuite suite : suites){
            Map results  = suite.getResults();
            for(Map.Entry entry : results.entrySet()){
                ITestContext itc =   entry.getValue().getTestContext();
                for (ITestResult tr : itc.getFailedConfigurations().getAllResu
lts()){
                    log ("Failed Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getFailedTests().getAllResults()){
                    log ("Failed Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedConfigurations().getAllRes
ults()){
                    log ("Skipped Config: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

                for (ITestResult tr : itc.getSkippedTests().getAllResults()){
                    log ("Skipped Test: " + name(tr));
                    log (asList(tr.getThrowable().getStackTrace()));
                }

            }
        }
    }

    public void log(java.util.List trace){
        for (StackTraceElement ste : trace){
            String s = ste.toString();
            if (s.startsWith("sun.reflect.NativeMethodAccessorImpl")) {
                log("\n-------------------------------------\n");
                return ;
            }
            log("\t" + s);
        }
    }

    public void log(String s) {
        System.out.println(s);
    }
}

Shards in your Latte

Phil — Sun, 20 Jan 2008 22:40:08 +0000

Here are the slides from my "Shards in your Latte" PJUG presentation on January 15:

javasharpedges.pdf

The Josh Bloch / Bill Pugh puzzlers talk from JavaOne given at Google. The Elvis example comes straight from here, and they do a better example of describing it than I did.

The Java Puzzlers book.

A couple of things that came up at the talk:

There's no version of String.replaceAll that takes a Pattern. I said I thought there was without thinking for more than a second about it.
Until the most recent version of Groovy, it didn't have a classical C-style 'for' loop. There was a lot of disagreement when I mentioned it, and I should have been more precise when I said that as to not imply that Groovy had no 'for' loop constructs.

Please add anything you think I got wrong, or anything you found interesting.

Bubble, bubble, toil and Double

Phil — Sat, 08 Dec 2007 18:41:37 +0000

Inspired by Weiqi Gao's Friday Java Quiz this week, I came up with a few more curiosities of Java double. What do each of the calls to larger() print? Hint: this puzzle has nothing to do with type conversion loss of precision, it is only about the specified behavior of doubles.

public class Foo {
    public static void main(String[] args) {
        larger(0.0d, Double.MIN_VALUE);
        larger(1.0d, Double.MIN_VALUE*Double.MAX_VALUE);
        larger(1.0d, Double.MIN_VALUE*Double.MAX_VALUE*Double.MAX_VALUE);
        larger(1.0d/Double.MIN_VALUE, 1.0d/(1.0d/Double.MAX_VALUE));
        larger(1.0d/Double.MIN_VALUE, 1.0d/0.0d);
        larger(1.0d, Double.MIN_VALUE/Double.MIN_VALUE);
    }

    public static void larger(Double a, Double b){
        System.out.print(a + " or " + b + " ? ");
        if (a < b){
        System.out.println(b);
        } else if (a > b){
        System.out.println(a);
        } else {
        System.out.println("equal");
        }
    }
}

Highlight the hidden text below for the answers:

0.0 or 4.9E-324 ? 4.9E-324
1.0 or 8.881784197001251E-16 ? 1.0
1.0 or 1.5966722476277755E293 ? 1.5966722476277755E293
Infinity or Infinity ? equal
Infinity or Infinity ? equal
1.0 or 1.0 ? equal

String Theory

Phil — Mon, 03 Dec 2007 04:32:41 +0000

No, not that string theory. This string theory is testable. In fact, the moral of this story is this: don't make any assumptions about Java. The compiler or the VM. They do crazy stuff, like make your code run better. If you think you have an inefficiency in your code, test it to see. It could be that you're about to make your code less readable for no performance benefit.

The software I do at work has a code generation component, so there's a bunch of string concatenation and output. The main path for generation passes around a StringBuilder (which is 20% faster than the old synchronized StringBuilder) to multiple classes, which each concatenate code into it. Some of the code generation within each class still used string concatenation (+=) for building intermediate strings, and I knew we needed to convert it to use an SB. When I ran FindBugs on the codebase, this alone accounted for about 25% of the true positives.

The main uses in the code are things like:

String s = "";
for (Foo f: foos){
    s += "baz " + f.id + " { " + f.type + " } ";
}

StringBuilder sb = ...;
for (Foo f: foos){
    sb.append("baz " + f.id + " { " + f.type + " } ");
}

My original thought was that both of these were horrible and were creating n strings in memory for each concat, which where then immediately thrown away. However, after doing a few experiments and looking at the bytecode, I discovered a few things. First, the compiler converts Strings concat'ed using '+' into calls to StringBuilder.append. This means that there aren't any intermediate String objects generated in a call like "a+b+c", only a StringBuilder and the result String. The first snippet above is still bad, because to create new String to assign to s, it has to create two new immutable Strings, one with the value of the String to add and one with the result of the concat. The garbage collector then must clean up the SB, the String from the SB, and the old value of s. In the second snippet, the only intermediate objects that gets thrown away is a StringBuilder, but all of it's state is reused by the StringBuilder we keep. As the numbers below show, there's not that much difference between multiple chained calls to append and one call to append with the syntactic sugar of the '+'. Using the '+' syntax is much more readable, so this is the option I'm going with in my code.

The trivial moral for the story is never to use '+=', but using + inside an append is fine. The general moral is that from now on, I'm definitely going to measure, then change.

The results for various String concat methods:

#	test	code	server (ms)	client (ms)	notes
1	string + w/ vars	s += a + b + c	929	2388
2	string + w/ consts	s += "a" + "b" + "c"	902	2380	the compiler transforms this into 's += "abc"'
3	string + w/ SB	s += new StringBuilder().append(a) .append(b).append(c).toString()	915	2390	like bytecode of (1)
4	str bld +	sb.append(a + b + c)	9	14	common pattern
5	str bld + CTC	sb.append("a" + "b" + "c")	7	11	constants instead of variables (4)
6	str bld ++	sb.append(new StringBuilder() .append(a).append(b) .append(c).toString())	9	14	bytecode of (4)
7	str bld app	sb.append(a).append(b).append(c)	7	11	most 'proper' usage
8	MString append	sb.append(a,b,c)	8	12	impl. of "faster" interface

This is the example code for the tests. Note that closures would have been really useful here, since I wouldn't have had to duplicate this method (can't wait for 7!).

    public long runStringBuilderAppend(){
        long start = System.currentTimeMillis();
        try {
            StringBuilder sb = new StringBuilder();
            for (int i = 0; i < runs; i++){
                sb.append(a).append(b).append(c);
            }
            sb.toString();
            sb = null;
        } finally {
            if (gc) System.gc();
            return (System.currentTimeMillis() - start);
        }
    }

source: StringTest.java