I found this post useful for understanding problems with handling "poison messages" in message-driven beans:
The gist of it is that even if you think you caught an exception, your transaction still might roll back and cause the JMS message to be re-delivered. (The MDB is an EJB, which will default to container-managed transactions, equivalent to having REQUIRED on each method.)
There are two small caveats to add:
1. The specific issue with JPA is that certain persistence exceptions mark the transaction for rollback, even if the exception is actually caught. It matters not whether that JPA activity is happening directly in onMessage() or in a "sub-transaction".
2. Simply annotating another method in the same MDB with @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) does not actually create a separate transaction context. If you are calling a method locally within the same EJB, "this" refers to the object itself, while the transaction behavior lives in the EJB proxy wrapper. (See http://stackoverflow.com/questions/427452/ejb-transactions-in-local-method-calls) So you actually have to put this transaction in a separate EJB and inject it into the MDB with @Inject or @EJB annotations.
And now for the "WTF" part.
The thing that really surprised me is the retry behavior. If the MDB throws an exception, and the transaction rolls back as a result, Glassfish recognizes that there was an exception and will only re-deliver the message once. There is some setting somewhere that controls how many retries are attempted, I believe. (Haven't found it.)
If you catch an exception that marked the transaction for rollback (or the transaction was marked for rollback programmatically), the transaction still rolls back, and the message is redelivered. However, the rollback without exception does not fire Glassfish's retry counter, so you end up in an infinite loop.
Either way the solution is the same, but still -WTF!