People are sometimes confused about the differences between WebSphere ESB (built on top of WebSphere Application Server, and using the inbuilt WebSphere Platform Messaging JMS provider) and WebSphere Message Broker (which uses WebSphere MQ as a messaging engine). IBM sometimes describes the latter as ‘Advanced ESB’, but Message Broker is not a superset of the functionality in ESB. There is a good FAQ on the IBM website which clears up some of the confusion.
In general, Message Broker is designed for working primarily with WebSphere MQ, as well as having a larger set of nodes (or mediation primitives, in WebSphere ESB-speak). WebSphere ESB has a richer set of functionality with respect to SOAP and Web Services (Message Broker treats SOAP as plain XML). ESB is also built on SCA, which allows it work more easily with products such as WebSphere Process Server.
ESB and Message Broker can work together: using the MQLink functionality in WebSphere Platform Messaging, ESB can simulate an MQ server and exchange messages with MQ accordingly. They can be complementary and both form part of an SOA solution.