Bill Schneider, Software Architect

Handling poison JMS messages in Glassfish - infinite loop WTF?

2012-01-11T21:22:00.000-05:00

I found this post useful for understanding problems with handling "poison messages" in message-driven beans:

http://weblogs.java.net/blog/felipegaucho/archive/2009/09/24/handling-poison-messages-glassfish

The gist of it is that even if you think you caught an exception, your transaction still might roll back and cause the JMS message to be re-delivered. (The MDB is an EJB, which will default to container-managed transactions, equivalent to having REQUIRED on each method.)

There are two small caveats to add:

1. The specific issue with JPA is that certain persistence exceptions mark the transaction for rollback, even if the exception is actually caught. It matters not whether that JPA activity is happening directly in onMessage() or in a "sub-transaction".

2. Simply annotating another method in the same MDB with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) does not actually create a separate transaction context. If you are calling a method locally within the same EJB, "this" refers to the object itself, while the transaction behavior lives in the EJB proxy wrapper. (See http://stackoverflow.com/questions/427452/ejb-transactions-in-local-method-calls) So you actually have to put this transaction in a separate EJB and inject it into the MDB with @Inject or @EJB annotations.

And now for the "WTF" part.

The thing that really surprised me is the retry behavior. If the MDB throws an exception, and the transaction rolls back as a result, Glassfish recognizes that there was an exception and will only re-deliver the message once. There is some setting somewhere that controls how many retries are attempted, I believe. (Haven't found it.)

If you catch an exception that marked the transaction for rollback (or the transaction was marked for rollback programmatically), the transaction still rolls back, and the message is redelivered. However, the rollback without exception does not fire Glassfish's retry counter, so you end up in an infinite loop.

Either way the solution is the same, but still -WTF!

equals and hashCode on @Entity classes: just say no

2012-01-07T21:23:00.001-05:00

I've come to the conclusion that you should avoid defining equals and hashCode on JPA @Entity objects unless you have a really good reason to.

Doing it right is non-trivial, with all the gotchas and caveats. First, the logistics:

You can't use the primary key because multiple unsaved objects all have a null PK and would be "equal".
Using the PK with object identity as fallback means an object won't be equal to itself before and after being persisted, and hash-based collections won't work properly if the hash code changes during the collection's lifespan.
A unique business key could work, if that's an option - but you have to remember to maintain the equals and hashCode method if those properties or relations change. Also if that key isn't immutable, you have a similar problem as with PK + fallback above.
Creating a UUID in the constructor just to make equals and hashcode work is... yucky.
Often these methods are insufficiently covered by unit tests.

Second: there's the deeper question of what should "equals" mean in the context of a mutable business entity? What comparison makes sense is often context-dependent. Sometimes you might want to compare based on primary key, other times you might want to compare on some other property.

So what's the worst thing that happens if you just don't do anything, and take the default based on object identity?

Usually, you won't even miss these methods. The default implementation will work fine unless you're trying to compare objects loaded in two different persistence sessions/transactions,with either equals() or contains(). I've found this is the exception case - much of my collection manipulation in practice is manipulating objects all loaded in the same session to hand off to the view layer, so HashSet recognizes duplicates and contains() works just fine. And I almost never use an entity as a key in a hashmap.

The price you pay is more verbosity when you do need to compare objects between a session-scoped cache and the current request. You have to remember that obj1.equals(obj2) doesn't work, and needs to be replaced with obj1.getId().equals(obj2.getId()). Contains is a bit more verbose, and a utility method like containsById(collection, obj) might be helpful. Some people will say it's confusing that equals doesn't "just work" but I find it less confusing to be explicit about what you're comparing on - and less confusing than a broken or unmaintained equals() method.

Finally, if this were C#, we wouldn't even have this discussion. With closures and LINQ extension methods we would just say "collection.Where(obj => obj.id = foo.id)" and be done with it!

Related links:
http://community.jboss.org/wiki/EqualsAndHashCode

http://onjava.com/pub/a/onjava/2006/09/13/dont-let-hibernate-steal-your-identity.html?page=3

http://stackoverflow.com/questions/1929445/to-equals-and-hashcode-or-not-on-entity-classes-that-is-the-question

http://stackoverflow.com/questions/1638723/equals-and-hashcode-in-hibernate

https://forum.hibernate.org/viewtopic.php?t=928172

http://www.ibm.com/developerworks/java/library/j-jtp05273/index.html: "For mutable objects, the answer is not always so clear. Should equals() and hashCode() be based on the object's identity (like the default implementation) or the object's state (like Integer and String)? There's no easy answer -- it depends on the intended use of the class... It is not common practice to use a mutable object like a List as a key in a HashMap."

http://burtbeckwith.com/blog/?p=53

Code coverage - unintended benefit

2012-01-03T15:53:00.000-05:00

Unit test coverage is not a panacea. You can reach 100% unit test coverage without a single assertion about outputs or business logic, in which case the tests aren't that useful.

Or are they? Just the act of executing every single line and branch of code does provide one important value: it demonstrates that you can show what inputs or conditions trigger which parts of the code.

More importantly, the act of getting there helps reveal sections of code and conditionals that you might not even need. Having a coverage target also gives you incentives to stay "clean" and avoid gratuitous not-null checks or try/catch blocks, two pet peeves of mine. (I've seen too many not-null checks that looked defensive at first but in reality just kicked the can down the road, further obscuring the real problem.)

The other benefit I've found is that a coverage goal encourages refactoring that you should be doing anyway, because well-factored code is easier to unit test. For instance I had one JSF/JPA application where I was accessing JPA EntityManagers directly in the JSF managed beans. This made unit tests on the managed beans annoying because I had to mock out EntityManager.createQuery and dozens of Query.setParameter calls for each JPA action. By pulling the JPA actions into a separate DAO layer, I could just mock a single call to myDAO.getStuff(arguments). Plus, after isolating the DAO, I could then write an integration test on the DAO hooking up to a real database.

Other related links:
http://www.wakaleo.com/blog/316-code-coverage-as-a-refactoring-tool
http://codebetter.com/patricksmacchia/2009/06/07/high-test-coverage-ratio-is-a-good-thing-anyway/

Primefaces AJAX callbacks: onstart vs. onclick

2012-01-03T15:38:00.001-05:00

I just learned the hard way that onstart and onclick are not the same thing.

In particular, a "return ..." has very different semantics in both cases.

Consider this code:

<p:commandLink action="#{bean.method}" onstart="return func()" ...>

If "func()" return false, this code will abort the AJAX request and bean.method() won't get called.
If "func()" returns true, the AJAX request processes.
If you replace onstart with onclick, the AJAX request will abort even if func() returns true.

That's because the Primefaces puts the code to generate the AJAX request in the onclick handler, pre-pending your code from the p:commandLink onclick before it. If your code returns, the AJAX request never gets sent.

Atlas Debugged: The Fountainhead, YAGNI and "clean code"

2011-12-30T22:32:00.000-05:00

The last time I re-read The Fountainhead, I felt like many of the ideas in the book could be useful for software developers. It seemed like there are many parallels between the values demonstrated by the book's hero, Howard Roark, and good development practices.

Many of the ideas dovetail nicely with some Agile development principles, especially the various "keep it simple" ones like YAGNI and DRY (along with DRY's cousins,"once and only once", and "three strikes and refactor"). Roark's designs are driven entirely by purpose, function and constraints, with an open disdain for non-functional or ornamental additions. For example, about the house he builds for Austen Heller, Roark says:

“Every piece of it is there because the house needs it – and for no other reason… You can see each stress, each support that meets it... But you’ve seen buildings with columns that support nothing, with purposeless cornices, with pilasters, moldings, false arches, false windows… Your house is made by its own needs. The others are made by the need to impress. The determining motive of your house is in the house. The determining motive of others is the audience.”

Keeping things simple and focused is hard work, just as agile development and YAGNI is not an excuse for sloppiness. If done correctly, it should be quite the opposite. In Roark's designs, “Not a line seemed superfluous, not a needed plane was missing. The structures were austere and simple, until one looked at them and realized what work, what complexity of method, what tension of thought had achieved the simplicity.” [emphasis added]

This made me think of the quote in Uncle Bob's Clean Code: "Learning to write clean code is hard work... You must sweat over it. You must practice it yourself, and watch yourself fail. You must watch others practice it and fail."

There were some other ideas I'm hoping to examine in more depth:
- right tool for the job - using technology idiomatically vs. legacy patterns with new technology
- the architect as a hands-on practitioner (vs. ivory tower)
- leveraging innovations - new methods, technologies etc.
- professional satisfaction / motivation
- interactions with business stakeholders, "people skills" and organizational politics
- making best of bad situations - looking for best possible solution even if you don't agree with the business problem to be solved

Primefaces global AJAX events

2011-11-12T09:04:00.001-05:00

You can use jQuery global AJAX events with PrimeFaces to refactor behaviors that appear on multiple components. A good example is if you have a data grid with multiple p:commandLinks and other controls that execute different methods and re-render the grid, and need to run the same onComplete in all of them.

For example, if you start with something like this:

<h:panelGroup id="grid">...</h:panelGroup>
<p:commandLink update="grid" actionListener="#{bean.update1}" onComplete="updateGrid()"/>
<p:commandLink update="grid" actionListener="#{bean.update2}" onComplete="updateGrid()"/>
<p:commandLink update="grid" actionListener="#{bean.update3}" onComplete="updateGrid()"/>

You could pull the updateGrid() statement out into an AJAX listener like this

jQuery(document).ajaxComplete(function(e, xhr, opts) {
  $response = jQuery(xhr.responseXML);
  if ($response.find("div[id$='grid']").length > 0) {
     updateGrid();
  }
});

By using ajaxComplete and parsing the XHR response object, you can see which DIV was going to be impacted by the partial update.  That way, if you had some other AJAX controls (say, an autocompleter) that you didn't want to trigger the updateGrid() function, you could filter that out.  Another option would be to set global=false on the specific Primefaces components that you don't want to fire the global jQuery ajaxComplete event.  The Primefaces autocompleter doesn't support this in Primefaces 2.x but does in 3.0.

Primefaces p:ajax and jQuery AJAX events

2011-11-08T20:13:00.002-05:00

All of Primefaces' JSF components use the jQuery AJAX engine, which means you can catch global AJAX events with $(document).ajaxStart and $(document).ajaxStop.

Standard JSF components like h:selectOneMenu will also go through jQuery when using a p:ajax facet (vs. standard f:ajax).

Conversely, jsf.ajax.onEvent does not appear to work with Primefaces components as it does for f:ajax.

IE9 and strict JSON checking

2011-11-08T13:08:00.000-05:00

Got an error from IE9 complaining about "invalid character". Turned out this was because the JSON response string had control characters (newlines) in it. These need to be escaped.

Other browsers will let you get away with it, but IE9 is particularly strict and jQuery doesn't check the JSON response string before handing it off to JSON.parse.

Similar issue found here:https://mail.mozilla.org/pipermail/es-discuss/2010-June/011420.html

Serialization, class hierarchy and preserving sessions

2011-10-26T08:39:00.000-04:00

If you have a class that implements Serializable, superclass fields do NOT get serialized unless the parent class also explicitly implements Serializable.That bit me the other day with session-scoped objects and preserving sessions across restarts, using Glassfish 3.1.1. Glassfish will complain if any objects in session scope or their properties don't implement serializable, but if those objects extend some parent class, the parent class fields are silently ignored and end up being null on session restore. So the source of the problem wasn't immediately clear, like it would have been if individual session objects or properties within them weren't serializable.

Quick and dirty SSO with LTPA

2011-10-22T19:54:00.001-04:00

If you have WebSphere application server in your environment, it is in fact possible to decode the "LtpaToken" cookie in code for quick-and-dirty SSO with non-WebSphere apps.

The main reason you might want to do this is if you have a portal-like application on WebSphere and want to link to other applications on different non-WebSphere servers. This is only useful if WebSphere is your main point of entry.

Here's how you do it:

Export the LTPA encryption key to a file from WebSphere using the admin console. You provide a passphrase and a filename.
Find the "com.ibm.websphere.ltpa.3DESKey" value in the exported file. This is the encrypted key.
Base64 decode the above key and decrypt with 3DES, using the passphrase provided. The decrypted value is the actual key for decrypting LTPA tokens.
Take the "LtpaToken" cookie, base64 decode it, and decrypt it with the key. The legacy LtpaToken cookie (which you can get with "interoperability mode") is encrypted with 3DES; the newer LtpaToken2 cookie uses AES.
Convert to String and parse. The string looks like "values%expiration%signature" where the expiration is a standard UNIX timestamp, which you should use to ensure the token is still valid; and the values somewhere will contain the user DN (e.g., uid=user,ou=company,dc=com).

The Alfresco codebase contains a good example of how to do this in Java

http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/modules/quickr/source/java/org/alfresco/repo/lotus/ws/impl/auth/LtpaAuthenticator.java

Relevant fragments of code:


    private static final String AES_DECRIPTING_ALGORITHM = "AES/CBC/PKCS5Padding";
   private static final String DES_DECRIPTING_ALGORITHM = "DESede/ECB/PKCS5Padding";

    private byte[] getSecretKey(String ltpa3DESKey, String ltpaPassword) throws Exception
    {
        MessageDigest md = MessageDigest.getInstance("SHA");
        
        md.update(ltpaPassword.getBytes());
        
        byte[] hash3DES = new byte[24];
        
        System.arraycopy(md.digest(), 0, hash3DES, 0, 20);
        
        Arrays.fill(hash3DES, 20, 24, (byte) 0);
        
        final Cipher cipher = Cipher.getInstance(DES_DECRIPTING_ALGORITHM);
        
        final KeySpec keySpec = new DESedeKeySpec(hash3DES);
        
        final Key secretKey = SecretKeyFactory.getInstance("DESede").generateSecret(keySpec);

        cipher.init(Cipher.DECRYPT_MODE, secretKey);
        
        byte[] secret = cipher.doFinal(Base64.decodeBase64(ltpa3DESKey.getBytes()));
        
        return secret;
    }
    
    private byte[] decrypt(byte[] token, byte[] key, String algorithm) throws Exception
    {
        SecretKey sKey = null;

        if (algorithm.indexOf("AES") != -1)
        {
            sKey = new SecretKeySpec(key, 0, 16, "AES");
        }
        else
        {
            DESedeKeySpec kSpec = new DESedeKeySpec(key);
            SecretKeyFactory kFact = SecretKeyFactory.getInstance("DESede");
            sKey = kFact.generateSecret(kSpec);
        }
        Cipher cipher = Cipher.getInstance(algorithm);

        if (algorithm.indexOf("ECB") == -1)
        {
            if (algorithm.indexOf("AES") != -1)
            {
                IvParameterSpec ivs16 = generateIvParameterSpec(key, 16);
                cipher.init(Cipher.DECRYPT_MODE, sKey, ivs16);
            }
            else
            {
                IvParameterSpec ivs8 = generateIvParameterSpec(key, 8);
                cipher.init(Cipher.DECRYPT_MODE, sKey, ivs8);
            }
        }
        else
        {
            cipher.init(Cipher.DECRYPT_MODE, sKey);
        }
        return cipher.doFinal(token);
    }
    
    private IvParameterSpec generateIvParameterSpec(byte key[], int size)
    {
        byte[] row = new byte[size];
        
        for (int i = 0; i < size; i++)
        {
            row[i] = key[i];
        }
        
        return new IvParameterSpec(row);
    }
// How to use it:
byte[] secretKey = getSecretKey(ltpa3DESKey, ltpaPassword);
byte[] ltpaTokenBytes = Base64.decodeBase64(ltpaToken.getBytes());
String token = new String(decrypt(ltpaTokenBytes, secretKey, DES_DECRIPTING_ALGORITHM));
// or
String token = new String(decrypt(ltpaTokenBytes, secretKey, AES_DECRIPTING_ALGORITHM));

Bizarre Glassfish JSF/EL performance issue

2011-10-12T21:42:00.002-04:00

I found a performance issue with JSF on Glassfish 3.1.1 (and prior) in an unexpected place: a seemingly innocuous line of code like

<h:outputText rendered="#{request.requestURL.indexOf('page') ne -1}" .... />

It turns out that invoking methods through EL expressions (new in EL 2.2) triggers creating a new ExpressionFactory, which in turn calls findResource/getResourceAsStream - a file I/O operation.

These expressions in the "rendered" attribute are particularly bad because they get executed multiple times in various parts of the JSF lifecycle.

See:
http://java.net/jira/browse/JAVASERVERFACES-2223
http://java.net/jira/browse/UEL-19

On the plus side - jvisualvm totally rocks. I am shocked by how easy it was to get started with for tracking down these issues and it was right there in $JAVA_HOME/bin all along.

Infinite loop in Glassfish exception logging

2011-10-11T21:33:00.001-04:00

When Glassfish catches a ServletException, it calls getRootCause recursively on each successive unwrapped, nested exception by reflection. This is fine for wrapped ServletException and JspException, which have well-defined semantics for this method, but creates a big problem if you have an application-defined exception that happens to use the same "getRootCause" name with different semantics.

So I created a Glassfish bug report:
http://java.net/jira/browse/GLASSFISH-17390

Related original bug in Tomcat, since fixed:
https://issues.apache.org/bugzilla/show_bug.cgi?id=39088

2011-09-25T09:45:00.000-04:00

Connecting MicroStrategy 8.x Web SDK to a MicroStrategy 9.x Intelligence Server (iServer) doesn't work. That in itself isn't a surprise. What is surprising, though, is that the failure persists and then prevents the 8.x Web SDK from working with an 8.x iServer like it's supposed to on subsequent connections, resulting in an error "The required encryption level is not supported in this release."

Apparently, this is a result of initializing "static" fields on the first session creation, such that those values get stuck for the lifetime of the classloader. In this case, the connection to 9.x sets some of these values in ways that are incompatible with 8.x. If the first session creation is to an 8.x iServer, then there's no problem - attempting to connect to 9.x will break but won't prevent subsequent connections to 8.x.

This is an unusual edge case, to be sure. I only encountered it during development in a heterogeneous environment, with a custom portal that connects to multiple iServers.

solved windows 7 sleep problem

2011-09-23T22:23:00.001-04:00

Sleep mode was grayed out on new Dell laptop (Latitude E6420) on Windows 7.

The fix was to disable the legacy VGA driver, which came up with a yellow warning icon in Device Manager ("device cannot start").

Simply installing the latest Intel Graphics drivers did not fix the issue - the legacy VGA driver had to be explicitly disabled.

502 Proxy Error solved

2011-09-17T19:40:00.002-04:00

I was getting mysterious, intermittent proxy errors using Apache mod_proxy to talk to an older version of Glassfish (2.x). It turned out the root cause was size of HTTP headers as cookies were accumulating, which caused the underlying app server to close the socket without a response.

A similar phenomenon was documented here:
http://kenai.com/jira/browse/KENAI-2727

The fix was trivial: increase the "receive-buffer-in-bytes" setting in domain.xml to 8192 bytes.

Finding it was a pain, because the older version of Glassfish had a bug such that the underlying IOException that aborted the request never got logged anywhere!
http://java.net/jira/browse/GLASSFISH-5181

JSF + JPA without EJB or Spring

2011-09-07T21:37:00.001-04:00

I've had success using JSF 2.0 with JPA on Glassfish 3.0.1, without using Spring or EJB for managing middle-tier services or DAOs. (I actually do like Spring - the goal in this case was to minimize dependencies, moving parts and learning curve, rather than to avoid any particular technology.)

A similar approach was detailed here:
http://www.swview.org/blog/best-way-use-jpa-web-tier

Using only the JSF @ManagedBean and @ManagedProperty annotations, along with the JPA @PersistenceContext, you can refactor business services or DAOs into other JSF managed beans that are injected into the one exposed directly to the UI.What this looks like in practice:

orders.xhtml:

OrderBean.java:

@ManagedBean
public class OrderBean {
  @ManagedProperty(value="#{orderDao}")
  private OrderDao orderDao;
  private List orders;
   
  @PostConstruct
  public void init() { 
    this.orders = orderDao.getOrders();
  } 
  public List getOrders() { 
    return orders;
  }
}

OrderDao.java:

@ManagedBean
@ApplicationScoped
public class OrderDao {
   @PersistenceContext(...)
   private EntityManager em;
   
   public List<Order> getOrders() { 
     return em.findAll(Order.class);
   }
}

In this example, the DAO (OrderDao) is just another JSF managed bean, but not actually referenced directly from a page, only from another managed bean. This approach lets you isolate the logic tied to EntityManagers from the rest of your "normal" JSF managed beans, and makes it (marginally) easier to unit-test your managed beans because you can mock the DAO at a higher level instead of mocking the entity manager.

This does NOT buy you declarative transactions or any of the other good stuff you get with Spring or EJB. (You can inject a UserTransaction with @Resource, and call it explicitly, though.) So it works best for simple apps with mostly read operations and basic single-object CRUD transactions.

Also, with Java EE 6 CDI, all this may become moot because @Named and @Inject annotations effectively blur the line between all the different managed bean flavors (EJB, JSF, Spring/POJO), although I haven't found a good way to replace JSF @ViewScoped without buying into Seam.

WebSphere "invalid Oracle URL specified" error

2011-09-03T10:59:00.001-04:00

Another unhelpful WebSphere error, this time with an assist from Oracle.

This happened to me when I configured my JDBC data source with the default wpdbJDBC_oracle JDBC provider, using the XA datasource (OracleXADataSource), and used the "container managed" J2C authentication alias instead of "component managed". The WebSphere admin console will successfully test the connection, but when you use it in a web application, it will fail with this "Invalid Oracle URL specified" error. It was so hard to track down because it made me focus on the JDBC URL, which wasn't ever the problem. It never occurred to me that the admin console and the web applications would somehow be getting connections and signing into Oracle differently, which tricked me into thinking that my configuration was really ok when it wasn't.

For the record, the web application was just doing a straight JNDI datasource lookup without any resource-ref mapping in web.xml, using the same JNDI name as bound in the server.

Also, changing to a different non-XA JDBC provider using plain OracleConnectionPoolDataSource resulted in "invalid username/password".

When I changed the datasource to use the component-managed alias instead, and restarted, everything worked.

WebSphere Portal error - "Puma requested entity type Group from VMM but received Entity"

2011-08-30T10:24:00.001-04:00

Got this error when trying to upgrade from WebSphere Portal 6.0 to 7.0.

com.ibm.portal.puma.SchemaViolationException: EJPSG0053E: Puma requested entity type Group from VMM but received Entity.

This sets a record for one of the most unhelpful errors ever. It can be caused by a bad entry in LDAP - in my particular case, it was a few groups whose objectClasses (groupOfUniqueNames and top) had somehow been saved with base-64 encoding instead of as plain text, although the values themselves when decoded were correct. The same exact LDAP directory was also working fine with WebSphere 6.0.

One of IBM's pages implies that the configuration itself is broken - when in my case, the configuration was perfectly fine and the problem was a bad entry in LDAP. The error gives no help tracking down the broken entry.

https://www-304.ibm.com/support/docview.wss?uid=swg21419580

Other IBM resources imply that the specific configuration issue is making sure that the group and user LDAP object classes are unique - i.e., don't use "top". In my case, I was doing pretty standard stuff - groupOfUniqueNames and inetOrgPerson.

http://publib.boulder.ibm.com/infocenter/wpzosdoc/v6r1/index.jsp?topic=/com.ibm.wp.msg.doc/messages/com.ibm.wps.puma.resources.Messages.html

https://www-304.ibm.com/support/docview.wss?uid=swg1PK80507

Javascript, getYear and cookie expiration

2011-08-26T20:12:00.000-04:00

Chrome and Firefox both return an offset from 1900 for date.getYear() instead of a four-digit year (pre-Y2K, a two-digit year). So new Date().getYear() returns 111 instead of 2011.

Old news, right? Well, I've been getting away with this for a while and not realizing it, in logic to expire cookies, and didn't notice until switching to Chrome.

The cookie expiration function was something like this

Date d = new Date();

d.setYear(d.getYear() - 1); // last year = 110

document.cookie = "COOKIE_NAME=; expires=" + d.toGMTString();

Firefox was happily setting the cookie's expiration to the year 0110, which was wrong but still accomplishing the end goal of expiring the cookie.

Chrome, on the other hand, mangled toGMTString to only print the three digits "110" and was not interpreting the cookie as expired.

Of course, the better question is why write this code yourself anyway, when jQuery, Dojo, etc., all have their own cookie APIs.

Javascript associative array and iteration order

2011-08-24T09:31:00.000-04:00

I just noticed that some browsers, like IE9 and Chrome, don't preserve insertion order of keys when iterating with a for-in loop ["for (key in array)"]. Although technically Javascript never guaranteed preserving order in a for-in loop, so many popular browsers did anyway that I was taken by surprise.

The Chrome behavior is especially odd - it preserves order of insertion for non-numeric keys, but iterates numeric keys in numeric order first. It also coerces numeric strings to numbers.

This code example illustrates

var foo=new Object();



foo["111_"] = 1;

foo["222_"] = 2;

foo["333_"] = 3;

foo["444"] = 4;

foo["555"] = 5;

foo["666"] = 6;



var str = "";

for (var key in foo) { 

  str = str + foo[key] + " ";

}

document.write(str);

In Chrome this prints "4 5 6 1 2 3". In IE 7, IE 8, and Firefox, it prints "1 2 3 4 5 6".

This behavior (particularly the coercion of numeric strings) has been tracked as a bug in Chrome - but unclear whether it will ever be fixed
http://code.google.com/p/chromium/issues/detail?id=37404

ECMAScript spec says "order of enumerating... is not defined"
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf

commons-logging classloader pain

2005-06-18T07:52:00.000-04:00

I recently learned about commons-logging's classloader behavior the hard way when moving applications that use log4j from Tomcat to WebLogic. I had naively put commons-logging.jar in my WebLogic system classpath to emulate Tomcat's behavior, and things subsequently broke. This web site provided a very detailed and erudite explanation:

http://www.qos.ch/logging/classloader.jsp

Basically, it boils down to weird subtlety in commons-logging's behavior. If commons-logging.jar is in the system classpath in a J2EE environment, the "current" or "thread context" classloader for each individual webapp contains WEB-INF/lib, but commons-logging itself is loaded in the system (parent) classpath. commons-logging looks for log4j using the thread-context classloader and therefore will "see" log4j in WEB-INF/lib, but tries to actually load it in the system classpath, and breaks with a NoClassDefFoundError.

Various solutions:

Only put commons-logging-api.jar in the system classpath, as Tomcat does. (I didn't realize there were two different variations of the JAR at first. I know, RTFM.) This stripped-down commons-logging will pretend that log4j doesn't exist. Downside: logging output from third-party APIs like Hibernate, Digester, etc. will go who-knows-where, but not the same place as your application's log4j output.
Put log4j in the system classpath too. This works, but has the bad side effect that all applications will share a single log4j configuration and this makes logging unmanageable.
Put commons-logging.jar in your WEB-INF/lib directory. I'd heard bad things about this, especially if you want to deploy the same WAR in Tomcat. But for whatever reason it worked fine with Tomcat 4.1.31 and WebLogic 8.1. So this may be my preferred solution.

Anyhow, if this was confusing to someone who has been doing J2EE professionally for 5-6 years, I can imagine how confusing it must be to someone just trying to do this for the first time. The BileBlog puts it a lot less charitably.

cost of supporting old browsers

2005-06-04T07:17:00.000-04:00

I've begun to question the wisdom of standards for websites that require support for older browsers like Netscape 4. Backwards compatibility is all well and good, but I feel like often times clients--government ones in particular--forumulate these requirements without any real cost-benefit analysis.

Keeping support for Netscape 4, IE 3, etc. has a very real cost, with respect to CSS and DHTML/Javascript one-offs, or having to pass on new patterns like AJAX. But many statistics show that fewer than 1% of all browsers are Netscape 4--even going back a year or two. (See here for an example.) Most people are using IE 6.

On the other hand, a commercial client I visited had done a very careful cost-benefit analysis on operating system support--they knew exactly how many of their users were still using Windows 98 and how much it would tick them off if they were forced to upgrade. So they might be stuck supporting it, but at least they could justify the decision.

So, before making requirements to support Netscape 4, clients should do their homework, and ask themselves what the benefit is--is anyone really still using it and can't upgrade? And how much extra time and money are the developers going to spend on workarounds to support these old browsers, compared to the value delivered?

Wireless adventure

2005-02-05T16:46:00.000-05:00

I must have bad luck with my home wireless network. I had a D-Link DI-514 wireless router, which worked great until I got a new IBM Thinkpad X40 with the built-in Centrino. After various firmware upgrades, configuration changes, etc., I could not get the two to work together--I was getting roughly 30% packet loss and 500 ms ping times, with the notebook right next to the router antenna.

So I got a new DI-524 which had a big Centrino-verified logo on the front. But, after unpacking it and hooking it up, it won't talk to my cable modem (also a D-Link product, a DCM-200) and the "WAN" link light doesn't even come on.

I take the thing back to the store, bring home another DI-524 *and* a Linksys BEFW11S4 just to be safe. Sure enough, they *all* have the same problem with the WAN link light. What are the odds?

After some googling and a call to D-Link tech support (who was way more knowledgeable than I expected) it turns out that this mass-market consumer equipment doesn't do a good job of auto-sensing 10- vs. 100-mbit ethernet, and my old cable modem is only 10mb. After a firmware upgrade it all worked.

The design lesson here is the "principle of least astonishment"--I'm somewhat knowledgeable about network stuff, but so far every piece of 100mbit ethernet equipment I've used doesn't just break if the other end is only 10mbit. So when that link light didn't come on, I immediately assumed the unit was defective, especially when there was no documentation of this issue. I could have saved myself (and Best Buy's returns counter) a whole lot of trouble if the problem were more clear.

displaytag sorting/paging performance vs. native SQL

2005-01-15T07:55:00.000-05:00

displaytag is an easy-to-use JSP tag library for printing out HTML data tables with sorting and paging. But, there's a catch. In an enterprise application, collections are the result of some database query, so implementing sorting and paging entirely within the presentation layer is not going to be as efficient as doing both natively in the database with ORDER BY, LIMIT and OFFSET (assume PostgreSQL syntax). Instead, displaytag expects the entire collection to be in memory, displaying only the necessary ones, and sorting the Java collection itself.

I ran some tests to see exactly how much this affects performance. On a non-scientific test (YMMV) with a 250 row result set, sorting with displaytag added an 34% overhead over a native ORDER BY. There was a much more dramatic difference for paging, though. On a test where I loaded 250 rows from the DB and displayed only 25 (pagesize=25), displaytag took 141% longer on average than a "LIMIT 25" which only returns the 25 rows to actually be displayed. The overhead factor will scale with the total number of rows in the original, unlimited query.

So, displaytag clearly is not as efficient as sorting and paging in native SQL. But is it enough of a difference to matter? Well, it depends on your application--it's a classic tradeoff between performance and elegant design. Separation of concerns says paging results conceptually belongs in the presentation layer; in a layered J2EE application, you can push sorting/paging down through the business and persistence layers, but it isn't pretty.

My personal opinion, though, is that performance normally shouldn't be enough of a factor to dissuade you from using displaytag for sorting/paging results. If your query returns so many rows that you can't afford the overhead from the undisplayed rows, then you may not be thinking about paging the right way--paging should be thought of as purely a UI layer construct to save the user from scrolling or downloading a large HTML document, not to save the database from working too hard. How meaningful is it to jump from page 1 to page 47 out of 62 anyway? If your query returns more than, say, 300 rows (15 pages with 20 rows each), you probably should think about making the user provide additional search criteria first rather than blindly paging the output.

What about the other option--coupling the presentation layer directly to the database? It's easy to imagine an extension to displaytag that takes a SQL query rather than a Java collection, and handles sorting/paging natively by dynamically appending ORDER BY and LIMIT/OFFSET clauses. (Indeed, the MS toolset seems to actively encourage this pattern.)

This is fine for prototyping or simple apps that don't have much business logic beyond the basic CRUD transactions. But, this can rapidly become unmaintainable--not only are there many more places to touch if the DB schema changes, but you are also at risk of introducing bugs by accessing tables directly and potentially bypassing domain logic (business rules) tied to particular fields. Still, it may be worth considering tihs as a strategy for hand-optimizing queries with special performance requirements that outweigh maintainability, especially if the query is more relational than object-oriented in nature.

In summary, displaytag is a good, simple choice for many applications that need HTML tables with sorting and paging. There may be a performance hit, but a clean, maintainable design often is more
important.

USB flash drives and Linux

2005-01-09T08:48:00.000-05:00

I've had to set up a USB stick with Linux recently. It's fairly easy if you know the right magic.

Here are some links to useful resources:

Post on ExtremeTech.com
Flash memory HOWTO on ibiblio

Essentially it boils down to adding a line to /etc/fstab and then mounting /dev/sda1 or /dev/sdb1 on some mount point (directory). You don't strictly need to edit /etc/fstab, but if you don't you will need to be root in order to write anything on it. Here's the magic for /etc/fstab to allow non-root to get read/write access:

/dev/sda1 /mnt/usbstick vfat        rw,user,noauto 0 0

(assuming your mount point is /mnt/usbstick.)

If that doesn't work, check /var/log/messages and see if your stick is on /dev/sdb1 instead.

I'm looking forward to it being as easy to use USB sticks under a default Linux distro as it is under Windows, though. I hate to say it, but under Windows, they "just work."

Avoiding Anemic Domain Models with Hibernate

2005-01-01T08:36:00.000-05:00

One of Hibernate's most under-appreciated features is its ability to persist private fields. This feature is useful for avoiding what Martin Fowler calls the Anemic Domain Model anti-pattern, where domain objects (entities) are reduced to "dumb" record structures with no business logic. In an Anemic Domain Model, you lose all the benefits of OOP: polymorphism, data hiding, encapsulation, etc.

The Anemic Domain Model may have originally evolved from EJB CMP, which requires any persistent field to be accessible directly with a public getter/setter. Developers using POJO frameworks like Hibernate often duplicate the same pattern, though, simply replacing the entity beans with POJOs.

This is not just an academic discussion; this has real consequences for the quality of a codebase. (Academically, this is part of the OOP-RDBMS "impedance mismatch"--in particular, that there is no distinction between a setter/constructor call that actually mutates/constructs an object and one that is merely incidental to materializing an existing object's state from persistent storage.) Let's say you're developing a system for issue tracking with a business rule like "anyone can create a ticket or change its status, but only managers can raise it to 'critical.'" A fragment of an Issue object might look like this (some detail omitted to focus on encapsulation/data hiding issues):

public class Issue {

private String m_status;

public String getStatus() {

  return m_status;

}

public void setStatus(String newStatus) {

  if (newStatus == STATUS_CRITICAL && !getCurrentUser().isManager()) {

    throw new SecurityException("critical.requires.manager");

  }

  m_status = newStatus;

}

}

This looks great until you realize that setStatus(STATUS_CRITICAL) is also going to be called from the persistence layer in materializing an existing Issue that is already critical, not just when making an explicit change through the UI workflow. Since anyone can view any issue, SecurityException will be thrown when a non-manager tries to view an issue that is already critical. We immediately recognize that the persistence layer needs a way to get "privileged" access to set the underlying field directly, bypassing business logic.

The typical workaround is to give up encapsulation and move the business logic into the corresponding service layer object (e.g., stateless session bean) for issue transactions:

public class IssueManager {

public Issue findIssueById(Long id) ;

public Issue newIssue(... fields ...) {

  // begin TX

  // ... setup new issue

  if (status == STATUS_CRITICAL && !getCurrentUser().isManager()) {

     throw new SecurityException("critical.requires.manager");

  }

  issue.setStatus(status);

  // ...

  // commit TX

}

public void changeStatus(Long id, String status) {

  // begin TX, load issue

  if (status == STATUS_CRITICAL && !getCurrentUser().isManager()) {

     throw new SecurityException("critical.requires.manager");

  }

  issue.setStatus(status);

  // commit TX

}

}

Now, two real consequences are apparent. First, giving up encapsulation leads to cut-and-paste programming, violating the "don't repeat yourself" principle; this increases the risk of error of the business rule not being cut-and-paste again somewhere it's needed. Second, you lose polymorphism; it is now very difficult to have a subclass of Issue with slightly different business rules. (For example, maybe the main Issue has no restriction on setting status, but a specific type of issue has the critical-requires-manager rule.)

It's true that you could have two separate sets of getters/setters in the Issue itself, one that applies business logic and one that allows direct access and is only used by persistence. This would address the polymorphism issue. But if that direct accessors are also public (as EJB CMP requires) then you still lose data hiding; nothing prevents your service layer/transaction scripts from calling these methods directly.

If you're using Hibernate, though, there is a very elegant solution. Hibernate is effectively "privileged" by manipulating bytecode, so it can touch private fields directly. Hibernate gives you two options in the above scenario:

You can have two separate bean-style properties linked to the same underlying field, one with private getters/setters and the other with public. The private methods access the underlying field directly, and the public ones apply business rules. This is the preferred approach, but has the downside of verbosity, plus you have to use different property names in HQL (private) and everywhere else (public).
Hibernate can also persist fields directly by using the "access" attribute on @hibernate.property and so on. The upside is that this is more concise with only a single public bean-style property, but using access="field" requires the field name to exactly match the private instance variable name; this won't work if you have some kind of Hungarian naming convention like "m_foo". You can do something like access="MyFieldAccessor" where MyFieldAccessor is a custom class implementing net.sf.hibernate.property.PropertyAccessor, implementing your naming convention (mapping bean property names to member var names) but that requires extra effort.

There are other uses for this feature in Hibernate:

Primary keys are generally supposed to be immutable by normal business logic, set only within the persistence layer. So, "setId" methods can almost always be private or protected.
Collections getters and setters can also be kept private, to preserve data hiding (prevent rep exposure). Otherwise, when business logic can manipulate a collection directly, it's difficult to enforce business rules on the collection elements, or even to ensure the elements are of the correct type. (The latter may partially be addressed by generics in Java 5 and/or Hibernate 3.)

I believe JDO also instruments classes at runtime to get similar privileged access to persistent fields.

PostgreSQL on cygwin: "Bad system call"

2004-12-22T21:02:00.000-05:00

I was getting this "Bad system call" message from PostgreSQL 7.4.x on cygwin. It was working earlier, and I thought I had done everything right--cygserver was up and running, so I didn't know what was up. ipcs gave the same error.

Turns out I had forgot the magic word: "CYGWIN=server". The first time I installed PostgreSQL (and read the docs), I had just set the var on the command line (CYGWIN=server pg_ctl start ...) and never put it in my profile. Easy enough to fix.

PostgreSQL performance of "where exists"

2004-12-15T22:34:00.000-05:00

Today I was looking into the performance of a PostgreSQL query with a "... where exists (select 1 from ...)" subquery:

select foo_id from foo

where exists (select 1 from bar where bar.foo_id = foo.foo_id)

I was surprised to find out that this query actually ran faster when I restructured it with a SELECT DISTINCT and a JOIN:

select distinct(foo_id) from bar

join foo on bar.foo_id=foo.foo_id

Some references on the web I've found suggest that EXISTS is the preferred way to write the above query in general. Because it's a boolean condition, in theory the database needs to scroll fewer rows because it can stop as soon as the first match is found; and the DISTINCT can be expensive if the results from the join version would not have been unique.

An ancient PostgreSQL mailing list post indicates that rewriting the query as a JOIN may be faster than EXISTS in PostgreSQL, because the join can take advantage of indexes while EXISTS does a nested loop. But, then again, I'm still using PostgreSQL 7.3.x, and EXISTS handling may well have been improved in 7.4.

"Client CVS Branch" anti-pattern

2004-12-11T12:07:00.000-05:00

Scenario: Your team developed a custom application for Client A. The application is generally useful, so it gets re-sold to Client B. Client B wants some customizations, which are at first superficial (CSS, images, etc.), and the client expects a quick turnaround. So, you need a way to store Client B’s new version of the app in source control somehow, and you take the first approach that comes to mind: you create a branch of the original, and make the customizations for Client B on the new branch.

(Without loss of generality, I’m assuming you’re using CVS; Subversion and Perforce have different branch/merge models but they basically boil down to the same thing. But if you’re using PVCS or VSS, or worse, not using source control at all, you have bigger problems which I’ll save for some other time.)

At first, this works pretty well. But over time, you sell the same app to more clients, and they are each asking for more substantial new feature development. The strategy starts to break down, causing a whole series of problems:

- Bug fixes have to be explicitly merged into each client’s branch individually.

- Since the code in each client branch diverges over time as different things are added or changed in one or the other, merging bug fixes results in more manual conflict resolution. (The same thing goes for new features.) This also makes for more re-testing of the same thing.

- There is a potential for wheel reinvention. If you develop a new feature for Client C, and Client D asks for a new feature that is “close but not quite” the same as Client C’s, it may be developed independently twice rather than building it once in a way that accommodates both sets of requirements.

- You fail to realize potential economies of scale of in support or maintenance that you should get from having a single solution, since each client effectively has their own one-off version.

The problem stems from the lack of a well-defined “trunk” in source control that provides the common baseline functionality. Instead, each client’s version was branching off of another client’s branch (Client A’s) rather than from a common trunk. So there was no way to nail down which part of the code stays constant for all clients.

Here's a few ways to solve this problem:

- Have each client’s version be a branch from a common trunk, and have the discipline to make as much functionality in the trunk configurable at deployment/runtime as possible (the later the binding, the better). That way, you increase the percentage of code that all clients have in common, and establish a common baseline version that multiple clients share. Also, there will then be a well-defined process for upgrading a client’s branch to a new version of the baseline “core” code, and many fewer post-merge conflicts to resolve manually.

- Most teams won’t actually have the discipline to consistently put in the extra effort to make new features configurable. So you can take this a step further: give each client is own separate repository that contains only the customizations for that client, then apply those customizations as a patch against the common baseline version. This will force you to think about whether any given code should be common and shared across all clients or whether it is a customization.

- There is also a management aspect to solving this problem: make sure someone is accountable for the entire solution as deployed for all clients, not just each individual client’s project. When you’re only accountable for your own client, you’ll inevitably take the path of least resistance to keep your own client happy and not see the bigger picture of delivering for all clients more effectively (this is actually rational behavior according to game theory).