Andrew Ferrier

Marc Andreessen discussed the history of programming in a recent podcast, noting the shift from machine-targeted languages to human-targeted languages. Although he seems to think that the change was fairly sudden (for him, it was started with Java in 1995), and he spends a surprising amount of time discussing PHP, which seems to be his favourite example of an easy-to-use language, his general point is still well founded.

It’s certainly interesting to see how languages like C and C++ are fading, their place being taken by Java. This is simply due to economic incentives: why spend expensive programmer time solving problems than can be solved by a cheap machine? Java is in danger of becoming overly feature-laden, but it still has a single important strength over its predecessors – simple dynamic memory management – no more explicit object creation/destruction. This does for memory what filesystems have done for disk. It’s hard to see just how much this has done for development speed and robustness until you compare it to what went before. PHP, Python, and so on, have the potential to do more, particularly for the new breed of web-based applications.

Andreessen also briefly discusses Web Services. He certainly seems to believe in Web Services – as he notes, the majority of major languages have good bindings for them now. I can see good things ahead for Web Services as the annoying details are sorted out by the market: see the doc/literal discussion from Andre Tost’s article I linked to a few days ago for an example. I look forward to seeing how this pans out.

In the world of business-oriented software, we use the phrase ‘business logic’ a lot. In my particular area (WebSphere integration products), a typical pattern is to delegate ‘technical logic’ to mediation flows in WebSphere ESB, and to put ‘business logic’ in process flows in WebSphere Process Server. It doesn’t have to be that way, but that’s what many people encourage, as it seems to neatly match what the software can provide.

But what do we mean by the phrase ‘business logic’? Ultimately both business logic and technical logic come down to the same thing; instructions executed by the machine. I can implement both in mediation flows, Java, Perl, assembler, or shell scripts. So why the distinction? It really depends on one’s perception.

For example, let’s imagine we are developing an integration system for a peanut factory, and we are developing a system to handle the orders. A typical way we might encourage someone to use WebSphere ESB is at the boundary, to convert between the protocol used by the web-based bulk order system (say SOAP/HTTP) to the one used by the back-end production system (say JMS). The distinction between these isn’t important to the business; it doesn’t matter what format the message is in; it just has to be in a certain format because we need some integration and don’t want to change everything.

By comparison, we might encourage someone to use a business flow to model (say) the approval of orders of peanuts. The process might follow a number of steps, some of which might involve human intervention, to modify the order (message) accordingly – i.e. adding ‘approved’ or ‘denied’. This would be based on the size of the order, that customer’s past history, whether they were salted or (urgh) dry roasted – all things that are at the business level – things that we find interesting (assuming we like peanuts).

Therefore, let’s characterise the difference as follows:

• Technical logic: Stuff we don’t want to do, but need to in order for everything to work.
• Business logic: Stuff we do want to do – the reason we’re doing all this. We can not only decide on how to implement this (as we can with technical logic), we can also decide not to implement it all, or to implement different logic.

Therefore, these two types of logic are at different abstraction layers – the technical layer we’d love to make as simple as possible and hide away (and I’d like to think WebSphere ESB does a good job of helping IBM’s customers do that), whereas the business layer we find interesting and want to spend some time on. I mentioned something similar before as a best practice – I could rephrase that best practice here as: ‘Spend as much time as possible on the business logic, and as little time as possible on the technical logic’. It seems sensible, then, that to keep them separate (and there are plenty of patterns for doing just that) is a natural thing to want to do.

I’d love to hear some comments from my consultant colleagues on this: do you think this distinction is important? Does it impact the way you work day-to-day?

Java is a pretty robust language for the objectives it seems to set itself – being a clean and easy-to-learn object-oriented language – although the slippery slope towards featuritis is very apparent in 5.0. The automatic garbage collection, in particular, is a godsend for someone migrating from C++.

But there are still plenty of little niggles that could be rectified:

1. Make the structure of the program dependent on indentation, like Python, and get rid of the curly braces everywhere. It makes sense.
2. Get rid of the primitive data types (int, long, etc.) at the source level. Auto-boxing in Java 5.0 makes them less annoying, but they still aren’t necessary – the lack of a primitive string datatype makes that clear.
3. Have an explicit modifier for package-private/friendly visibility, for consistency, and make it mandatory to specify a visibility modifier. These things make the code less ambiguous for those unfamiliar with the rules.
4. Remove the one-class-per-source-file restriction (in theory this is for public classes only, but because of this, most people use a separate file for each class). It makes for huge and unwieldy build and deployment structures. C++ shows how it can be done.
5. Get rid of the syntatic inconsistency between interfaces and classes. Methods in interfaces are always abstract as they cannot have a body. Java allows the abstract modifier to be missed out, however. This can be confusing.
6. Make /* */-style comments nestable. It is very irritating to comment out a block, but have to do it in bits because of a ‘real’ comment block that’s part of the code. Colour-coding text editors mean that there shouldn’t be any confusion when doing this. It would also make //-style comments largely redundant.
7. Merge the concepts of list and arrays, as Perl does. This allows for a far greater degree of expression in the language, and reduces some of the OO waffle necessary in Java when dealing with lists.

This kind of stuff is almost always about personal preference, so it’s unlikely that you’ll agree with all of the above. But I’d love to hear your comments anyway. What little things annoy you about Java?

The Sapir-Whorf hypothesis basically states that there is a relationship between the language that a person uses and the way they think about the world. Although it’s controversial, many linguists believe there is at least some truth in it: as Wikipedia says, ‘The opposite extreme—that language does not influence thought at all—is also widely considered to be false’. The theory has implications, such as that the value of improving one’s vocabulary or learning another language are even greater than they would be otherwise.

The hypothesis has also been extended to programming languages. Essentially, the theory here is that certain types of language are more suited to solving certain problems, and programmers who aren’t aware of these types may not be able to solve some problems in the most effective way. I think this is borne out by empirical evidence: there is a noticeable difference in the way those who have been trained in declarative programming (such as functional and logic programming) solve problems, even in when writing in traditional procedural languages (for example, C), from those who are only trained in those procedural languages. This is important for some types of problem, such as writing compilers or parsers, which are well suited to declarative programming.

Almost all software, whether it be public domain, open-source, or commercial, is not just released once. Typically it goes through many versions, changing (and hopefully improving) each time, sometimes adding features, sometimes removing bugs or problems, sometimes introducing new ones. This change is normally managed (at least for larger software systems) using two or three elements:

• Version control systems (also called source control or library systems). These are typically used to keep track of different versions of source-code, enabling previous versions to be retrieved, different parts of the software to be merged together, and so on. CVS is a well-known open source example.
• Bug tracking systems (bugs are sometimes called defects). These are used to keep a list of all the outstanding bugs and details about them. Bugzilla is a well-known open-source example.
• Feature or requirement tracking systems. These are used to track new features that need to be added to the software (wherever they may come from).

In some cases, these systems may be integrated, in other cases not (integrated systems are normally more useful: for example, you can easily ask the question: ‘when this bug was fixed, what code changed?’).

Currently, commercial software companies often follow something akin to a waterfall model and treat features as long-term work items and bugs as short-term (as an aside, the original inventor of the waterfall model didn’t believe in its effectiveness). This is a risky game to play if your competitor can implement new features before your model allows you to. It’s likely that as commercial software becomes under increasing pressure from open-source competitors, often using nimble development methods such as agile programming, commerical software production companies will need to turn around production of new versions with new features faster than before to survive.

One of the mismatches occurs in treating bugs and features independently. Bugs take time to fix. Features take time to implement. In a sense, both are actually customer requirements, even if the customer hasn’t asked for them. Bugs customers won’t notice shouldn’t be fixed, and bugs they will notice will become a requirement even if they haven’t noticed them yet. Features either the customer has asked for or you think they may want should be implemented. Features that only you want are a waste of time. Bugs that your customer thinks are bugs normally are, even if you want to call them a feature.

So this leads to my conclusion (and not everyone agrees): bugs and features are actually the same type of thing. They are both customer requirements and should be managed the same way, triaged the same way, and developer time assigned to them in the same way.

Many open-source projects already treat features and bugs like this. Firefox’s Bugzilla, for example (in common with most Bugzilla installations), tracks features with a severity of ‘enhancement’. This probably isn’t ideal; features shouldn’t be treated as second-class citizens compared to bugs. But it’s a step in the right direction. This way, developers of Firefox see bugs and features as being of the same kin; lack of features cause some customers just as much pain as lack of bugfixes.

This issue is far from settled, and there are strong opinions out there that disagree with me. But it is clear that everyone involved in software development should be aware that this is not a solved problem.

Working with WebSphere ESB recently has got me thinking about synchronous and asynchronous communication mechanisms. I began to wonder about a mechanism that’s more familiar to most people – text messaging – and how it plays a dual role in usage. Let’s recap first on two other communication mechanisms that more obviously fit into one category or the other (perhaps ironically, a modern mobile phone can do all of these, including text messaging):

• Phone call – synchronous. You are engaging in an ongoing conversation with the other party. The flow of conversation goes back and forth at a high rate, and you have to be present when the phone call is occurring to participate (ignoring voicemail, recordings etc.).
• Email – asynchronous. You can send someone a message and it doesn’t matter when they read it – it could be in 2 minutes, 5 hours, or 3 days. The message can get to them via a number of ‘store-and-forward’ hops, rather than directly. When and if they reply, the same rules apply.

Text messaging, in many ways, is technically similar to email. It has different protocols, is typically used with different types of software, and has different formats. But it’s still store-and-forward and is still technically asynchronous.

However, I would assert that in many (although not all) situations, text messaging is, from a usage point-of-view, synchronous. The stories about teenagers who rack up £1000s in phone bills or develop RSI from text conversations across the classroom attest to that. Although they are using a mechanism which is by nature asynchronous, they are actually able to use it in a semi-synchronous way because it is reliable and fast. Whilst most adults’ use of text messaging isn’t quite so dramatic, text ‘conversations’ of 3 or 4 messages in fast succession each way aren’t uncommon (I have done this, although rarely, because I have large thumbs and find phone keypads awkward – don’t laugh!). Also, though, text messages are still used asynchronously in many situations (I don’t have any statistics on this but would hazard a 50:50 ratio as a not unreasonable estimate).

In many ways, this isn’t anything new – it has always been possible to emulate synchronous messaging with fast asynchronous messaging (arguably, this is what voice-over-IP systems such as Skype do, for example). But what I do find striking about text messaging is its dual usage role – sometimes synchronous, sometimes not. Often the difference is only in the perception of the sender – some people expect prompt replies, others perhaps not for days, and it all depends what one’s sending anyway.

This morning I was writing a test plan document. It contained lots of technical nitty-gritty and detail, but a lot of it was the same stuff repeated over and over again. It was very tedious to write (and hence undoubtedly error-prone), but necessary. I kept thinking how useful it would be if I could automate the process. I knew it was just a little too complex, and contained a few too many exceptions, though, that writing a program to do it would be more trouble than it would be worth.

This led me to thinking about ‘near-human’ tasks. Perhaps we can classify tasks into three board areas:

• Automatable. These are tasks that can easily be programmed, and where it makes sense to do so once you have a computer. For example, figuring out the prime numbers between 1 and 1 million is far faster on computer than by hand (including the time to write the program).
• Non-automatable. These are tasks that almost certainly cannot be programmed, even with the current state of AI, and are often open questions: ‘How do I prevent wars?’, ‘Is global warming real?’, ‘Which way should I design my new product?’, ‘What’s the best colour for a car?’, etc.
• Near-human, or semi-automatable. These are tasks (such as mine above), that feel like they should be automatable, but anyone who has any software engineering experience knows that they will spend more time writing the program to do it than doing the task (although only by a factor of up to 10, say).

It wouldn’t surprise me to discover that near-human tasks are an already-understood concept, probably in the field of AI or some other computer science theoretical field (possibly something I should already know!), and that I’m reinventing terms here. But maybe they an area that we can focus on more in order to overcome some frustration with the menial tasks that annoy us?

One possible example, although more automatable than my original one, is that of detecting spam. Relatively good spam filters now exist for email, with ~1% false negative rate and next-to-no false positives. These spam filters have been probably been technically possible for some time (although they still eat a fair amount of compute resources), but have only risen to prominence in the past few years because the rising tide of spam has made them necessary. They are still not perfect, which in my mind is what keeps them out of the fully-automatable category (the writers are always one step behind the spammers, and would have to do more work than is worthwhile, or maybe even possible, to get a 0% false negative rate), but they are getting closer.

What other examples of near-human tasks exist? What should the bright software engineers of tomorrow be working on to make our lives easier?

Audio and video players such as RealPlayer use read-ahead buffering on the client when streaming across networks to accommodate for temporary slow-downs or delays in network traffic. Most of them also use a form of pre-buffering, in which a certain amount of data is read before playback starts (or when the buffer runs out). Some – such as RealPlayer – will also dynamically alter the bitrate being requested, if network performance drops for a period of time. This noticeably affects the quality of the audio/video.

There is a possible complement to the bitrate method – if minor network delays are suspected, and the buffer is moderately full, slow down the audio or video by a small amount (perhaps 3-15%). This would be mostly unnoticeable, as it’s possible to dynamically ‘correct’ the pitch in real time, but would slow down the rate at which the buffer is emptying, and reduce the likelihood of skips or pauses whilst the buffer is pre-filled again. It would probably have to be combined with the bitrate method, as that would have a more drastic effect on the data rate.

I’m not aware of any player currently available that does this (WinDVD can slow down/speed up DVD playback, with pitch correction, but as far as I know, this only applies to ‘local’ DVDs). I’d be interested to hear any thoughts on the theory or a possible implementation.