To recap, WebSphere ESB provides a bunch of re-usable mediations out of the box that you can use in your mediation flows to alter message content (XSLT mediation), filter on it (Message Filter mediation), and so on. However, it also provides the facility to create your own mediations, called ‘custom’ mediations, in mediation flows. Typically, you’d use these when the provided mediations can’t do something you want to do. They are SCA components implemented by a Java class, so appear in the assembly diagram for a mediation module (where you’ll see that the mediation flow component references them), but they also appear in the mediation flow itself as a mediation. They provide an execute method, which has a DataObject as a parameter (message coming in) and return a DataObject (message going out). This is where your logic goes. Normally you don’t need to worry too much about all of this, as WebSphere Integration Developer makes it easy to create a custom mediation in your flow like any other mediation, and creates a Java skeleton for you - all you have to do is define the method. You can’t specify any of your own parameters on a custom mediation, so typically you’d use these where you are doing something specific that you’re unlikely to re-use. Nigel Daniels has written an excellent article on how to implement custom mediations.
There is an alternative. You can create your own first-class mediation primitive that has its own icon in the WID mediation flow editor and can have custom properties. This is more work, as you are essentially creating a mediation equivalent to those supplied by WebSphere ESB, but it is thus suitable for re-using in many mediation flows. Confusingly, these are sometimes called custom mediations also (probably because they don’t really have a name). For example, you might write a generic mediation primitive that could do arithmetic on an incoming message, according to a set of parameters you specify. There is a document on the IBM website that describes how to create them.