Fortunately, underlying Firefly is a sqlite database, which can be queried using standard SQL syntax. This enables a technique of creating static playlists that are automatically re-generated periodically instead.

The prerequisites for the following technique are:

• Perl, with the File::Spec module (to convert from absolute paths to relative ones, which is what Firefly expects).
• The sqlite3 command-line interface.

The three commands that follow will create a standard .m3u playlist with the top 100 most-played songs from Firefly’s database, and another playlist with all the non-Podcasts added in the last month, ordered by the time they were added. Neither of these are possible using Firefly’s query language.

sqlite3 /var/cache/mt-daapd/songs3.db 'select path from songs order by play_count desc limit 100' | perl -nle 'require File::Spec; $_ = File::Spec->abs2rel($_, "$PLAYLIST_DIR"); print;' > "$PLAYLIST_DIR/Most-played songs.m3u"
 
MONTHAGO=$(perl -e 'use Date::Calc::Object qw(:all); $date = Date::Calc>now(); $date += [0,-1,0,0,0,0]; print $date->mktime();')
 
sqlite3 var/cache/mt-daapd/songs3.db "select path from songs where genre!='Podcast' and time_added > $MONTHAGO order by time_added desc" | perl -nle 'require File::Spec; $_ = File::Spec->abs2rel($_, "$PLAYLIST_DIR"); print;' > "$PLAYLIST_DIR/Music added in last month by most recent.m3u"

(obviously, if you use these, you’ll need to alter paths to suit, make sure the correct Perl modules are installed, remove line breaks to make it easier to read, etc.)

Firefly will read these .m3us if configured correctly during its next rescan, and use them as it would any other playlists. You can force a rescan with the following wget command:

wget --delete-after -q --http-user noone --http-password yourpasswd "http://localhost:3689/config-update.html?action=rescan"

Although not fully dynamic (they are not generated on request from the Soundbridge), if these commands are called from cron or similar, the playlist can be kept up-to-date ‘enough’.

• Always use integrated source-code libraries and bug tracking. This is something that CMVC and other systems do excellently, and up till recently was fairly poorly served by the open-source software community, although projects such as Trac are doing a good job of closing this gap. The ability to see what changes are associated with what bug is invaluable. Anything else is a recipe for mistakes.
• Get rid of all the excuses. There are only two valid reasons for permanently closing a bug: (i) it’s fixed; (ii) the developer disagrees that the change will improve the software for the user. Anything else isn’t OK. A corollary of my rule about bugs and features being the same thing is that bugs can’t be returned just because they are feature requests. All bugs that aren’t targeted for a release currently being worked on are still open bugs (just with a different target field). All bugs we don’t want to fix because they don’t seem sensible to fix right now (which could be marked as such) are still open bugs. All bugs in an external dependency are still bugs (the whole system doesn’t work). All bugs that can’t be recreated were seen once (assuming you trust your testers) and are still bugs. A good bug tracker is a database, and will let you see any subset of these any time you want, so no-one needs to get blamed unfairly.
• Make sure each bug only has a few panic fields. ‘Severity’, ‘Importance’, ‘Priority’, ‘Ease of Recreation’, ‘Impact on Customer’, ‘Impact on Developer’, ‘Impact on Tester’ are all ambiguous. Pick a maximum of two, preferably just one, and make sure everyone knows exactly what they mean. After all, only one field really matters – how much does this affect the user of our software? Everything else should be secondary.
• What Joel says about explaining off-buttons to uncles applies equally to the design of procedures around software development – everyone will think their exception to the process is vital until you’re drowning in exceptions. This relates more to conventions surrounding software development rather than meta-software itself, but it’s still relevant. Keep it simple – regular builds on a schedule everyone knows; keep everything in one place; reduce the number of parties required to make a decision about any change to the bare minimum. Scott Berkun has a lot to say about this in his excellent book The Art of Project Management.

Fundamentally, though, maybe none of the above will help. Meta-software is perhaps destined to suffer from featuritis more than other software precisely because usability is not so important for its userbase (in my experience, most developers don’t like bad interfaces, but can also cope with them). Only time will tell if developers will be set free.

Clay Shirky, in his essay A Group is Its Own Worst Enemy (also included in the same volume; the online copy is edited slightly differently from the printed one), notes how online (synchronous) discussions frequently descend into talk about sex – and that sexual banter is much more common in synchronous communication than asynchronous (how often have you flirted with someone over the phone compared to email? – please, no anecdotes in the comments section). I’m not a psychologist, but I assume that this has something to do with it being hard to retain the thrill of adult banter over the course of a (potentially lengthy) asynchronous discussion. The same arguments probably apply in a less dramatic fashion to non-sexual communication.

There’s a related observation to be made about the perceived economics of people’s time. In general, most folks implicitly value synchronous time as higher than asynchronous – if I ask advice of a mentor over a half-hour coffee, I feel more indebted to him than if he spends half an hour hour answering my email. I suspect the reasons are a combination of my having accurate information (I know exactly how long he spent drinking the coffee), the start-up and tear-down time (he actually took 5 minutes to get to the coffee shop), and knowing that I have his undivided attention (he wasn’t multi-tasking). Nevertheless, we still continue to rate synchronous time more highly than its opportunity costs compared to asynchronous time.

To relate the two assertions, wouldn’t you rather spend half an hour in person with your spouse / significant other / other politically correct phrase than an hour writing and exchanging emails with them? Synchronous communication has a strange attraction than its poor cousin doesn’t – despite all of asynchronicity’s time-shifting advantages. This is going to be a big challenge for a multi-time-zone world.

As the more capable e-mail spammers seem to be figuring out that anti-spam technologies are getting pretty good at filtering out the crap they send, normally using rule-based detection, Bayesian learning, or a combination of the two, a lot of spam now being sent out is image-based – and anti-spammers are now using OCR to fight back against this new tide.

As I’ve mentioned before, I have a huge spam problem on my personal e-mail account (~4,000/week) – due to a combination of bad luck and some foolish naivety at a few points – and so I have a fairly highly-tuned SpamAssassin installation running at home, with plenty of custom rules and plugins. I’ve seen a rising amount of image spam on it, so I decided to give FuzzyOcr, a plugin for SpamAssassin, a try. So far, the results are pretty impressive. FuzzyOcr uses the open-source gocr program as the engine, and ties it to with SpamAssassin and some logic. The OCR is fairly CPU-intensive, so unlike most SpamAssassin plugins, it only kicks in if the message is otherwise going to be below a certain scoring threshold. So far it has roughly halved the volume of spam that slips through into my inbox (previously ~40-50/day), which is a welcome improvement.

However, fun though they are as a technical challenge, technical approaches such as these always feel like fighting a losing battle. I might write a lengthier article on this at a later date, but I’d like to see ISPs take a far more hardline attitude with their peers that host spammers. There are also compelling economic solutions to the problem, mostly related to micro-payments for sending email. There are problems with those too (how do you roll them out gradually?), but you rarely see graphs of spam that have a downward trend – a solution to the spam problem would be most welcome.

• Mr. Google, please develop a podcast search engine. So much interesting content is now being released as podcasts (quick plug for my favourite: EconTalk), that it would be useful to be able to search them. All you have to do is invent a speech-to-text interpreter that actually works reliably. Simple. [Note: as I sometimes do, I wrote this post in advance of it being published. I've since discovered that such a tool already exists. However, I thought I'd leave the original prose here: Google, if you get one out soon, you could still corner the market]
• Mr. Google, please stop developing so many interfaces – and plug them all together. If I want to do an exhaustive search for something, I now have to search Google Web, Google Images, Google Groups, Google News, Google Video, Google Blog Search, Google Book Search, Google Scholar, and possibly others. This is not a good thing – you’re straying from the simple search you started with. Some of those searches do show up in the main search results, but you could do a better job of tying them together to show what I’m actually looking for. This could be a real competitive edge, especially since the basic searches that MSN and others provide are now actually quite reasonable.

Google still have an edge in providing what people want – for a company so technically-focused, they either have talented marketers or are just lucky. Please, Google, keep it up.

• Marc Hull, who talked about his project on Balancing simplicity and efficiency in web applications. Marc’s work focused on improving the development of stateful web applications, and in particular on object-relational mapping in Java, in an attempt to allow more straightforward persistence of objects to databases. This has always seemed to me to be an area lacking in usability and ease (see J2EE for plenty of examples), so anything that moves us closer is welcome.
• Matthew Sackman, who talked about his project Glint: Breeding Mobile Ambients with Actors (which won the IBM Project Prize for the best final year Individual Project). Essentially, Matthew seems to be attacking the area of concurrent and distributed computing, in order to improve its robustness against deadlock (and other concurrency problems). He has chosen to do this by writing a compiler for the GLINT language, which is based on an Actor model and is especially particularly suitable for modelling concurrent systems.
• Francis Russell, who talked about his project Delayed Evaluation and Runtime Code Generation as a means to Producing High Performance Numerical Software. Francis’s infrastructure shifts some code generation and execution to the runtime of a program (lazy evaluation). It does this by building up a DAG to represent expressions that are ‘should’ have been already evaluated. The expression it represents isn’t actually evaluated until it’s needed, which enables certain optimisations to be performed (which is useful, for example, in matrix arithmetic). The framework generates and executes the optimised code at runtime (and it also caches this generated code).

I had the chance to meet these folks briefly (Marc and Matthew had also been here previously, when they were part of the team from Imperial who won the Thinkpad Challenge). It was interesting to see some academic work for a change – whilst I’d never be able to make a career out of that, bringing academia and business together always seems to reap benefits.

I wish Marc, Matthew and Francis luck if they choose to develop their projects further.

Greg at IBM Eye has provided a brief recap of the many software domains in which IBM supports open-source. I think Greg’s analysis is too narrow – there are other benefits to open-source software apart from savings on purchase cost – some of them bogus and some not – but quality and maintainability are certainly side-effects of open-source software, and are worth having even if they’re hard to quantify.

I’m not sure I can comment that widely on the state of current SOA projects, and I would agree that SOA may suffer from similar management problems to that of object-orientation: if developers of SOA systems aren’t rewarded for saving time with a reuse strategy, they won’t be enthused to do so. This is an important part of any software project, and encouraging reuse is a best practice that shouldn’t be restricted to object-orientation or SOA.

However, whilst I agree that SOA has other benefits apart from encouraging reuse, I have a fairly high opinion of its potential in that respect. It’s important to understand what we mean by reuse. Reuse rarely means using an object or service as it is. There is often a mismatch between the interface offered by the service (object) being consumed, and the service (object) that needs to call this interface. Expecting anything else is unrealistic (even if future reuse plans are made). This is often solved using something like a façade pattern in object-oriented languages, and some form of mediation with services (such as that offered by WebSphere ESB). The latter is often easier, because there is a lower degree of coupling than inside a single programming language, and because programming code is not often needed, and this is why I believe SOA reuse is simpler – if done well. Of course, some work is still required, but this greater ease of reuse makes it a realistic strategy for more scenarios.

I would agree, however, that, as is often the case, the project management problems here are the greatest ones.

• Steve Loughran and Julio Guijarro, HP Labs. This presentation was about Smartfrog, a system deployment framework, which Steve and Julio were working on as part of a strategy for system testing. They demonstrated several examples of how the system might work in practice. Smartfrog looks pretty flexible, and I plan to spend some time looking into it. Frameworks for deployment have an inherent problem in catering to the wide variety of platforms, configuration mechanisms, deployment combinations and so on that are necessary in practice. Anything that gets closer to this is therefore welcome. Smartfrog also has the interesting property that the XHTML it produces as output is sufficiently well-formed that, although it has an embedded CSS stylesheet for presentation in a web browser, it can also be parsed as XML data without much effort, and thus act as a machine-readable data source as well. This might seem obvious to some folks, and I’m willing to bet it’s not the first time it’s been done, but it seemed novel to me.
• Robert Chatley (a fellow alumnus from Imperial College) and Tom White, Kizoom. Tom and Robert were talking about what they called literate functional testing. Essentially this involves creating tests, in this case written in Java, that use plain English for method names, variables, and so on. This means that once punctuation is stripped out, Java code – test assertions – become statements that are readily understandable by non-programmers (such as the business analysts in their organisation). Their framework will shortly be available on Google Code.
• Andreas Leitner, ETH Zurich: Andreas discussed automated testing using contracts. Essentially this means using a language which has pre-conditions and post-conditions on methods, and optionally assertions on objects. He has developed a testing framework called AutoTest, using the Eiffel language, which has these features built in (similar extensions are available for more mainstream languages such as Java). Once these restrictions are placed on a program, generated input can be used to determine whether methods behave as they should. A number of strategies are available for generating this data, and they are pluggable into Andreas’s framework. The simplest is of course to generate the data randomly, but other, more sophisticated strategies are available to improve coverage. Andreas stressed that this type of testing, which is essentially fuzz testing with intelligence, should be used to complement human-created unit tests, not to replace them.
• Goranka Bjedov, Google: Goranka explained some of the background to performance testing, including the differences between performance testing, stress testing, load testing, scalability testing, etc., and how to deploy and manage performance testing systems.
• The conference finished with 10 lightning talks. Subjects covered included jMock (a mock objects framework that complements JUnit), justifying automated tesing in financial terms, testing heresies, Yandex (the largest search engine in Russia, their share ~60%, Google’s ~6%), ‘Automated Testing: Why bother?’, automated tricks for manual testing, and the Perl-inspired Test Anything Protocol. I hadn’t seen lightning talks before, and I thought they were a fantastic idea – similar in some ways to the Straight 8 showings I wrote about a while ago – if you don’t like what you’re seeing, something else is coming along soon. More presentations should be like this.

My thanks to the guys at Google for hosting this conference, particularly for free. The original call for attendees was on their research blog. Some of the conference topics were quite academic and in-depth, but that provided a good constrast to the more practical topics. Google’s offices and facilities are also impressive – definitely worth visiting if you get the chance.

Google LTAC

It’s certainly interesting to see how languages like C and C++ are fading, their place being taken by Java. This is simply due to economic incentives: why spend expensive programmer time solving problems than can be solved by a cheap machine? Java is in danger of becoming overly feature-laden, but it still has a single important strength over its predecessors – simple dynamic memory management – no more explicit object creation/destruction. This does for memory what filesystems have done for disk. It’s hard to see just how much this has done for development speed and robustness until you compare it to what went before. PHP, Python, and so on, have the potential to do more, particularly for the new breed of web-based applications.

Andreessen also briefly discusses Web Services. He certainly seems to believe in Web Services – as he notes, the majority of major languages have good bindings for them now. I can see good things ahead for Web Services as the annoying details are sorted out by the market: see the doc/literal discussion from Andre Tost’s article I linked to a few days ago for an example. I look forward to seeing how this pans out.

But what do we mean by the phrase ‘business logic’? Ultimately both business logic and technical logic come down to the same thing; instructions executed by the machine. I can implement both in mediation flows, Java, Perl, assembler, or shell scripts. So why the distinction? It really depends on one’s perception.

For example, let’s imagine we are developing an integration system for a peanut factory, and we are developing a system to handle the orders. A typical way we might encourage someone to use WebSphere ESB is at the boundary, to convert between the protocol used by the web-based bulk order system (say SOAP/HTTP) to the one used by the back-end production system (say JMS). The distinction between these isn’t important to the business; it doesn’t matter what format the message is in; it just has to be in a certain format because we need some integration and don’t want to change everything.

By comparison, we might encourage someone to use a business flow to model (say) the approval of orders of peanuts. The process might follow a number of steps, some of which might involve human intervention, to modify the order (message) accordingly – i.e. adding ‘approved’ or ‘denied’. This would be based on the size of the order, that customer’s past history, whether they were salted or (urgh) dry roasted – all things that are at the business level – things that we find interesting (assuming we like peanuts).

Therefore, let’s characterise the difference as follows:

• Technical logic: Stuff we don’t want to do, but need to in order for everything to work.
• Business logic: Stuff we do want to do – the reason we’re doing all this. We can not only decide on how to implement this (as we can with technical logic), we can also decide not to implement it all, or to implement different logic.

Therefore, these two types of logic are at different abstraction layers – the technical layer we’d love to make as simple as possible and hide away (and I’d like to think WebSphere ESB does a good job of helping IBM’s customers do that), whereas the business layer we find interesting and want to spend some time on. I mentioned something similar before as a best practice – I could rephrase that best practice here as: ‘Spend as much time as possible on the business logic, and as little time as possible on the technical logic’. It seems sensible, then, that to keep them separate (and there are plenty of patterns for doing just that) is a natural thing to want to do.

I’d love to hear some comments from my consultant colleagues on this: do you think this distinction is important? Does it impact the way you work day-to-day?

But there are still plenty of little niggles that could be rectified:

1. Make the structure of the program dependent on indentation, like Python, and get rid of the curly braces everywhere. It makes sense.
2. Get rid of the primitive data types (int, long, etc.) at the source level. Auto-boxing in Java 5.0 makes them less annoying, but they still aren’t necessary – the lack of a primitive string datatype makes that clear.
3. Have an explicit modifier for package-private/friendly visibility, for consistency, and make it mandatory to specify a visibility modifier. These things make the code less ambiguous for those unfamiliar with the rules.
4. Remove the one-class-per-source-file restriction (in theory this is for public classes only, but because of this, most people use a separate file for each class). It makes for huge and unwieldy build and deployment structures. C++ shows how it can be done.
5. Get rid of the syntatic inconsistency between interfaces and classes. Methods in interfaces are always abstract as they cannot have a body. Java allows the abstract modifier to be missed out, however. This can be confusing.
6. Make /* */-style comments nestable. It is very irritating to comment out a block, but have to do it in bits because of a ‘real’ comment block that’s part of the code. Colour-coding text editors mean that there shouldn’t be any confusion when doing this. It would also make //-style comments largely redundant.
7. Merge the concepts of list and arrays, as Perl does. This allows for a far greater degree of expression in the language, and reduces some of the OO waffle necessary in Java when dealing with lists.

This kind of stuff is almost always about personal preference, so it’s unlikely that you’ll agree with all of the above. But I’d love to hear your comments anyway. What little things annoy you about Java?

The hypothesis has also been extended to programming languages. Essentially, the theory here is that certain types of language are more suited to solving certain problems, and programmers who aren’t aware of these types may not be able to solve some problems in the most effective way. I think this is borne out by empirical evidence: there is a noticeable difference in the way those who have been trained in declarative programming (such as functional and logic programming) solve problems, even in when writing in traditional procedural languages (for example, C), from those who are only trained in those procedural languages. This is important for some types of problem, such as writing compilers or parsers, which are well suited to declarative programming.

• Version control systems (also called source control or library systems). These are typically used to keep track of different versions of source-code, enabling previous versions to be retrieved, different parts of the software to be merged together, and so on. CVS is a well-known open source example.
• Bug tracking systems (bugs are sometimes called defects). These are used to keep a list of all the outstanding bugs and details about them. Bugzilla is a well-known open-source example.
• Feature or requirement tracking systems. These are used to track new features that need to be added to the software (wherever they may come from).

In some cases, these systems may be integrated, in other cases not (integrated systems are normally more useful: for example, you can easily ask the question: ‘when this bug was fixed, what code changed?’).

Currently, commercial software companies often follow something akin to a waterfall model and treat features as long-term work items and bugs as short-term (as an aside, the original inventor of the waterfall model didn’t believe in its effectiveness). This is a risky game to play if your competitor can implement new features before your model allows you to. It’s likely that as commercial software becomes under increasing pressure from open-source competitors, often using nimble development methods such as agile programming, commerical software production companies will need to turn around production of new versions with new features faster than before to survive.

One of the mismatches occurs in treating bugs and features independently. Bugs take time to fix. Features take time to implement. In a sense, both are actually customer requirements, even if the customer hasn’t asked for them. Bugs customers won’t notice shouldn’t be fixed, and bugs they will notice will become a requirement even if they haven’t noticed them yet. Features either the customer has asked for or you think they may want should be implemented. Features that only you want are a waste of time. Bugs that your customer thinks are bugs normally are, even if you want to call them a feature.

So this leads to my conclusion (and not everyone agrees): bugs and features are actually the same type of thing. They are both customer requirements and should be managed the same way, triaged the same way, and developer time assigned to them in the same way.

Many open-source projects already treat features and bugs like this. Firefox’s Bugzilla, for example (in common with most Bugzilla installations), tracks features with a severity of ‘enhancement’. This probably isn’t ideal; features shouldn’t be treated as second-class citizens compared to bugs. But it’s a step in the right direction. This way, developers of Firefox see bugs and features as being of the same kin; lack of features cause some customers just as much pain as lack of bugfixes.

This issue is far from settled, and there are strong opinions out there that disagree with me. But it is clear that everyone involved in software development should be aware that this is not a solved problem.

• Phone call – synchronous. You are engaging in an ongoing conversation with the other party. The flow of conversation goes back and forth at a high rate, and you have to be present when the phone call is occurring to participate (ignoring voicemail, recordings etc.).
• Email – asynchronous. You can send someone a message and it doesn’t matter when they read it – it could be in 2 minutes, 5 hours, or 3 days. The message can get to them via a number of ‘store-and-forward’ hops, rather than directly. When and if they reply, the same rules apply.

Text messaging, in many ways, is technically similar to email. It has different protocols, is typically used with different types of software, and has different formats. But it’s still store-and-forward and is still technically asynchronous.

However, I would assert that in many (although not all) situations, text messaging is, from a usage point-of-view, synchronous. The stories about teenagers who rack up £1000s in phone bills or develop RSI from text conversations across the classroom attest to that. Although they are using a mechanism which is by nature asynchronous, they are actually able to use it in a semi-synchronous way because it is reliable and fast. Whilst most adults’ use of text messaging isn’t quite so dramatic, text ‘conversations’ of 3 or 4 messages in fast succession each way aren’t uncommon (I have done this, although rarely, because I have large thumbs and find phone keypads awkward – don’t laugh!). Also, though, text messages are still used asynchronously in many situations (I don’t have any statistics on this but would hazard a 50:50 ratio as a not unreasonable estimate).

In many ways, this isn’t anything new – it has always been possible to emulate synchronous messaging with fast asynchronous messaging (arguably, this is what voice-over-IP systems such as Skype do, for example). But what I do find striking about text messaging is its dual usage role – sometimes synchronous, sometimes not. Often the difference is only in the perception of the sender – some people expect prompt replies, others perhaps not for days, and it all depends what one’s sending anyway.

This led me to thinking about ‘near-human’ tasks. Perhaps we can classify tasks into three board areas:

• Automatable. These are tasks that can easily be programmed, and where it makes sense to do so once you have a computer. For example, figuring out the prime numbers between 1 and 1 million is far faster on computer than by hand (including the time to write the program).
• Non-automatable. These are tasks that almost certainly cannot be programmed, even with the current state of AI, and are often open questions: ‘How do I prevent wars?’, ‘Is global warming real?’, ‘Which way should I design my new product?’, ‘What’s the best colour for a car?’, etc.
• Near-human, or semi-automatable. These are tasks (such as mine above), that feel like they should be automatable, but anyone who has any software engineering experience knows that they will spend more time writing the program to do it than doing the task (although only by a factor of up to 10, say).

It wouldn’t surprise me to discover that near-human tasks are an already-understood concept, probably in the field of AI or some other computer science theoretical field (possibly something I should already know!), and that I’m reinventing terms here. But maybe they an area that we can focus on more in order to overcome some frustration with the menial tasks that annoy us?

One possible example, although more automatable than my original one, is that of detecting spam. Relatively good spam filters now exist for email, with ~1% false negative rate and next-to-no false positives. These spam filters have been probably been technically possible for some time (although they still eat a fair amount of compute resources), but have only risen to prominence in the past few years because the rising tide of spam has made them necessary. They are still not perfect, which in my mind is what keeps them out of the fully-automatable category (the writers are always one step behind the spammers, and would have to do more work than is worthwhile, or maybe even possible, to get a 0% false negative rate), but they are getting closer.

What other examples of near-human tasks exist? What should the bright software engineers of tomorrow be working on to make our lives easier?

There is a possible complement to the bitrate method – if minor network delays are suspected, and the buffer is moderately full, slow down the audio or video by a small amount (perhaps 3-15%). This would be mostly unnoticeable, as it’s possible to dynamically ‘correct’ the pitch in real time, but would slow down the rate at which the buffer is emptying, and reduce the likelihood of skips or pauses whilst the buffer is pre-filled again. It would probably have to be combined with the bitrate method, as that would have a more drastic effect on the data rate.

I’m not aware of any player currently available that does this (WinDVD can slow down/speed up DVD playback, with pitch correction, but as far as I know, this only applies to ‘local’ DVDs). I’d be interested to hear any thoughts on the theory or a possible implementation.