My new notes program

I've been working in my lab on something new, or a new version of something old. First, of course, I need to give you some background.

I have a note-taking program I use every day that saves text files to Dropbox, based on hashtags I supply for each note. I also have several computers, some of which are offline sometimes, because I write on the train or have no internet access at work or just haven't switched on my desktop in a day or two. All of this combines to cause problems with the text files - namely, conflicting edits. Dropbox helpfully preserves each copy of the file and I can usually mash them back together again, but sometimes I can't, or it's a lot of work.

So I've written a new version of my note-taking program that doesn't suffer these problems. It uses a design called event sourcing which means that, instead of writing the text files directly, I write a record of the events and actions that occurred, like making a new note, editing an old one, attaching tags and so on. What this means for the program is no more conflicts, or at least not the same kind. Each event is written to its own file with a unique generated name, so they never conflict when Dropbox synchronises. Everything is always perfectly and effortlessly preserved.

The down side is that I am no longer working with plain text files. I can't just open up a text editor and change an entry, or delete it. I need to go through the proper channels to make sure the event stream is preserved. For the sake of never having another conflicting edit in my text files, though, I think this might be worth it.

Mokalus of Borg

PS - It's still in testing, but it's coming along very nicely. From the UI, you can hardly even tell that it's changed.
PPS - The library and pattern will be used on all my other similar programs from now on, too.

Tracking your non-disclosure agreements

It's not really a broadly-applicable problem, but some people might benefit from NDA tracking software. I see it on Wil Wheaton's blog every now and then: he gets to work on some really exciting thing, but has to sign a non-disclosure agreement to say he won't talk about it for a certain amount of time. He gets excited about it, though, and posts to his blog about how excited he is to be working on this top-secret thing that he can't tell anyone about, and then usually about how grateful he is to be able to work so regularly on such exciting (secret) things. Then months later, when the NDA expires, he can't remember what the post referred to, so he never gets to update it.

So the idea is this: when you sign an NDA, you enter the details into this tracking software, which stores its data locally, strongly encrypted (because gathering NDA data on a server on the internet would be like waving a big "HACK ME!" flag). It gives you a random unique ID - possibly even just a plain integer starting from 1 - so you can talk around it publicly and still know, yourself, what you meant, without revealing anything to anyone. It would look like "Hey, guys! I just wrapped up work on this really exciting project, but I can't talk about it here yet. I'll come back and update you when my NDA5 expires." Then every week or so you can come back to the app and check for expired agreements, search for any posts you've made about them, then update with the details you are now allowed to tell.

Mokalus of Borg

PS - I wish I needed this.
PPS - There's honestly not much to it, though. I guess even a spreadsheet would work.

Exercise app review: Zombies, Run!

I took the exercise app Zombies, Run! for a spin on the weekend, just through "episode 1". For the full experience, I used no music and had zombie pursuits switched on. On the whole, I like the app - I get the feeling that the story will unfold in an interesting way, and that's probably what I'm in for. Base building is not much of a draw to me, but I see the appeal for other people. As I said, I'm mostly in to see how the story plays out, and I'm keen to do more running to discover more of it. Which would be the whole point.

The feature that didn't get me were the zombie pursuits. The voice announced "Warning: zombies nearby" or something to that effect. Okay, I thought, better put on a bit more speed. They're zombies, though. No need to overdo it. Pretty soon, the voice announced "Warning: zombies 50 metres". Oh, okay, they must have started closer than I thought. Better speed up some more, to a fast run. I swear, before I'd gone another 10 metres, "Warning: zombies 17 metres". What? Alright, done with this. I break into a full sprint. "2 items dropped, zombies distracted". I slowed to a walk, exhausted.

This pattern continued a few times over my run, and it got tiresome (which I suppose is better than winding up dead in a real zombie pursuit). This did not seem to be helpful to me. How fast am I supposed to be going? My usual speed is 9.5 to 10kph. Granted, this weekend I was not at my best. I'm recovering from a cold and I haven't been for a proper run in a few weeks. Still, when the voice tells you "If you've got two legs and can go faster than a slow shamble, you should be fine, right?" it's demoralising to have three or four successful zombie attacks per run. I went looking for how to switch off that feature later and couldn't find it. I hope that doesn't mean I'm stuck with it.

Mokalus of Borg

PS - If I can turn zombie pursuits off again, I'll keep using the app for now.
PPS - If not, I'll start using my running time to catch up on podcasts.

Writing the right software

Writing software is easy. Writing the right software is hard. A lot of the time, that's the problem: teasing out the requirements from the client, and then separating the blue-sky dreams from what should actually be done. A lot of software projects, even if they're on track for budget and time get into trouble when the clients ask for more. Cowboy developers who grew up writing their own software on their own time for their own amusement can take a while to learn that "could" and "should" are different. That is, in a way, even more important than the correct architecture. Badly written software that still functions is a pain, but software written to the wrong requirements doesn't do anyone any good.

Mokalus of Borg

PS - You have probably encountered both kinds.
PPS - Older software can drift either way, depending on whether the requirements change over time.

Declarative programming for tomorrow

I strongly suspect that, in the future, there will be much less imperative programming and more declarative. That is, we will spend less time, as programmers, telling our computers "this is how to do this task" and spend more time telling our computers "this is what the task is". Defining the conditions around the task at hand is a powerful mechanism, and is very well suited to, say, manipulating large sets of data in one go, or running in massively parallel circumstances.

My honours thesis at university was written in a declarative language called Prolog. It was designed to do calculations on certain types of code conditions which, in time, could have become part of a compiler that would tell you if your real-time conditions were likely to be met by your code. Complicated stuff, and we expected it to function only if it had certain solid numbers to work with. However, because of the way Prolog works, when we fed it symbolic data instead of concrete data, it managed to swallow the whole thing and still produce results. In computing terms, this is like teaching a child basic arithmetic and finding out later that they've conquered algebra all on their own, based only on your arithmetic lessons.

That's another kind of power declarative programming has. It can sometimes go beyond your expectations in perfectly valid and logical ways. It didn't make any difference to my program if it was manipulating "x" or manipulating "2". They're all symbols at some level. We need that kind of power, natural parallelism and simplicity of expression to conquer tomorrow's programming problems. We might not be able to expand today's most popular languages to handle these problems, though.

Mokalus of Borg

PS - It's likely there will still be declarative parts of these new languages.
PPS - It's difficult to avoid them.

Mature software project management

I have a feeling and a hope that, as the software development industry gains maturity, it will also gain some better, more solid theories and practices of project management. We have had some false starts in this area, I think, and also some very good developments, too. New languages and new tools are always being advanced, enabling new features that let us set up some really solid patterns of architecture. However, I still run into as many bad managers as good ones - those who don't understand software projects in general, or those who don't understand programmers. The clients, also, underestimate how long it takes to get something done, so unrealistic expectations are set. There is as much bad software as good in the world, or probably even more.

Would it help to teach everyone to program as part of the school curriculum? Yes, of course. It doesn't mean it's something we really need to do, though. I'm sure there are plenty of accountants, pastry chefs, firefighters and jet pilots who wish everyone knew more about their jobs to better appreciate how hard they are and how to set realistic expectations. It's probably better to develop a specialised course - "Programming For Non-Programmers" - to instill some of the necessary skills and knowledge for people when they need it.

Mokalus of Borg

PS - I've made it as far as that title before.
PPS - Maybe someday I'll put together at least a hypothetical course outline.

Over-structured data

Data can be overstructured. That's as bad as too little structure, because it doesn't take exceptional circumstances into account. For instance, say you've decided that a postal address always consists of a street number (strictly numeric), a street name, a street type (eg "Road", "Street", "Avenue"), a suburb, a state and a postcode. Well, in this case, maybe you missed a street type in your list, which means a certain type of street (eg "Place", "Esplanade", "Boulevard") is excluded. Or what about unit numbers? They all share the same street address, but delivering to unit 7 is very different to delivering to unit 17. You get the point. You need to strike the right balance between structuring your data to give it meaning and consistency, and leaving it unstructured to allow for unforseen circumstances. It can be a tricky balance to strike.

Mokalus of Borg

PS - The worst part is that the right structure is always changing.
PPS - Everything is always changing.

Removal of software features

Software sometimes faces the removal of features as it goes on in life. This is especially true of large, complex software with lots of users. Maintaining features through upgrades costs time and effort in development and support. Sometimes the justification is made that this or that feature or option is being removed and replaced with a permanent setting based on how many people don't use it, or the way most people keep the setting. This stands to upset as few people as possible.

The problem is that it does upset some people, and if you have thousands or tens of thousands of users, you're going to be upsetting a large number of people, however small the percentage is. That can get lost in the analysis. If "only" 35% of your users sort oldest-first, but the majority keep their lists sorted in the default newest-first order, you may be tempted to set the sorting option to newest-first permanently. However, if you have 100,000 users, you're about to cause problems for 35,000 of them. Some will adapt. Some will leave for software that still does what they want. Are you willing to risk those 35,000 users for the sake of not maintaining a sort option?

Mokalus of Borg

PS - For me, a lot of my software has an audience of one.
PPS - And when it doesn't, removal of features is not up to me.

The Big Red Button

You know those locked plastic covers they have over the Big Red Button in movies, regardless of what that Big Red Button does? They need that same thing over the "Reply All" button in email programs. That action needs to have some real gravitas. Not a pop-up asking "Are you sure?", but the action itself impressing upon you that this is a big thing you are doing.

Mokalus of Borg

PS - It's going to be a fair bit more annoying.
PPS - And that's kind of the whole point.

Arimaa

I'm kind of interested in the way a game was created specifically to be difficult for computers but easy enough for a 4-year-old to play with standard chess pieces, called Arimaa. It's an interesting goal to push computer game-playing research further, and it seems to have worked. The game was invented in 2002 and then a computer won the yearly human-vs-computer tournament this year (2015). On the one hand, this might represent massive gains in processing power and algorithmic research since the game's invention, or it might represent an unforeseen weakness in the game design. Most likely, it's a bit of both.

My question is: what comes next? Does someone else design a new iteration of game that is really tricky for computers to play but comparatively easy for humans? Do we modify Arimaa to bump up the computer difficulty? When do we reach the point where "easy for a 4-year-old" also means "trivial for computers"? If it takes decades for computers to beat human grandmasters at chess, but only one decade for computers to beat human Arimaa masters, how much difficulty can we expect to squeeze out of a new iteration of game difficulty? Probably a year or two, I'd think, given the pattern.

Mokalus of Borg

PS - I'd love it if a computer designed a game itself to this goal.
PPS - Which, I guess, would make that game a kind of CAPTCHA.

What separates programs from queries?

Sometimes the line is quite blurry between what is a program (or a programming language) and what is an expression. I'm not even completely sure what to call the second one, actually, but after reading about "runnable tweets" written in Wolfram Language and seeing the graphical plot results, it seems to me that these programs are more like search expressions or queries. They require a very rich interpreter with access to a lot of very complex data sets and extremely heavy functions to manipulate them. It's difficult for me to think of something like "GeoListPlot[GeoEntities[Atlantic Ocean, "Shipwreck"]]" as a program. It might be meaningful and unambiguous when interpreted against a particular data set, and clearly Wolfram can produce results from it, so I'm not sure what is my objection. Perhaps that there is only one system in the world that can run it? Well, that would be true of any very specialised software. I've written expression parsers that only my programs use, but I never called the input a program.

One objection might indeed be the power of the functions available. You could write a "language" whose only function is "List The Monarchs of England In Chronological Order", but is that input a program, or is the interpreter the program?

Maybe it's a size issue. Because of the massive scope of the language, and its inherent complexity, every program written in it will be many orders of magnitude smaller than the interpreter itself. I think that's what's bugging me.

Mokalus of Borg

PS - Plus there's something about the native data sets.
PPS - It doesn't feel like big, real-world data sets should be considered part of a programming language.

Estimating software is really hard

Are software estimates valuable? It's hard to say. We, as programmers, feel that estimates are very difficult to obtain, usually inaccurate and therefore something of a waste of time. Managers, on the other hand, have a certain budget available and often a certain time frame, too. They want to know if a project will fit into those constraints or, if they are a particularly talented manager, what subset of the blue-sky-dream requirements can be completed within that budget and time. Estimates are the way budgets turn into project plans.

The problem is that software estimates are always a tricky beast. Software isn't like constructing a building. It's more like designing a building for a swarm of robots to build in seconds for no cost. All the effort in software is design effort, not construction, and design is the hard part to pin down.

When you design a building, you have some solid requirements to work from. It will be this tall, with this many floors, we want this many units or offices inside, and this many car parking spaces. That, in turn, leads to some other calculated requirements - to serve that many floors, you need this many lifts. Fire stairwells are this size and regulations require two of them. Concrete can support this much weight per unit surface area. You know what you're getting into.

Software, so far, doesn't have any of that, and might never do. Also, by the time you're done finding out how large the project is, that's part of the design work. If, for instance, you're trying to measure business software by the number of database tables required, then you need to find out what tables are needed and count them. Finding out what tables are needed, though, means gathering all the requirements and designing tables to store the relevant data and oh, look, now you've done a database schema to estimate how long it will take to develop a database schema. It's almost as if the nature of software rebels against being quantified before it is completed. Unfortunately, "I'll know how much it will cost when I'm done spending the money" isn't a satisfying or wise answer to the question of budgets.

So what can we do? Maybe skilled managers have better intuition about how large a project will be, based on some imprecise metrics. If I know we're dealing with five kinds of business data here, then we'll probably have (for instance) 50 tables (10 per data type) and 10 major processes to write (interactions between the major data types), and from there we can guess how long that will take (assuming we know our team well).

The best way, probably, is to say "This is what we want to get done. We have up to this amount of money, or up to this time to do as much as we can. How close might we get to perfection?"

Mokalus of Borg

PS - I have to plead ignorance on how other industries estimate their work.
PPS - Except sometimes when I saw managers give a gut-feel guess and everyone called it good.

Why I started using Pushbullet

For some time I was using Firefox as my primary web browser everywhere - at home and at work, on my netbook and also on my phone via the mobile version. I would often pull open tabs from one device to another via the Sync functionality, and it worked pretty well ... as long as it had synchronised by the time I wanted to pull the tabs around and I hadn't closed the source browser before the sync had finished.

Then my phone started misbehaving, I started a new job where Chrome worked better with the web proxy and the whole system fell apart. I looked briefly at XMarks, which is supposed to have cross-browser tab sync, but I couldn't get that working at all.

So I installed Pushbullet, based mostly on the fact that it was a channel available in IFTTT. Now, even though I'm using Firefox at home, Chrome at work and the stock Android browser on my phone, I can push tabs instantly from one device to any other, as well as send small files to and fro and get weather alerts via IFTTT. It's very handy. It was an adjustment to go from a "pull" mentality to "push", but not bad.

The one feature that doesn't quite work is sending SMS from my desktop - I can send texts just fine, and I know they are received because people respond, but there is no record of them going out at all. Not on my phone, not on the Pushbullet browser plugin. Other people have this problem with the feature and for some of them it works to go out and back in to the messaging app or restart their phones. That hasn't worked for me, but I'm not too concerned. I'm very happy with the other features and I plan to keep using them for some time.

Mokalus of Borg

PS - I'm very happy with the way another machine doesn't have to be online for a push.
PPS - I prefer my multi-machine software to allow that.

Peer-to-peer MMOs

Since City of Heroes shut down - or even since the announcement, really - I've wondered about how to make MMO games robust against that kind of existential threat. When such games become unprofitable, development ceases and the servers are shut down. That's just business. The obvious answer to that problem is to host the game itself on a peer-to-peer network of player machines. Then, no matter how much the game grows or shrinks, there's always enough server capacity.

However, running a game like that on a peer-to-peer network raises some other challenges. For one, there's the matter of trusting the server code and preventing cheating. If the players, technically, have access to all the server code running on their own machines for each other, there's no central, trusted arbitrator for tasks like random number generation and application of the rules. I've outlined before how some trust can be established between peers for generating random numbers, so it's possible it could be worked around, but it requires a lot more communication than an implicitly-trusted server does. It's also probably not the full story for everything that's needed for a trusted peer network of this type.

Still, I'd like to see it attempted, if only to know that, in the future, there's a definite way to save these games from destruction when they become unprofitable, or for smaller, niche games to get a leg up when they're starting out and can't afford dedicated server hardware.

Mokalus of Borg

PS - On the City of Heroes front, a new game called Valiance Online has started open public testing.
PPS - Which is a long way from a complete game, but more than I've seen in a while.

Snowdrift could fund free software

I really like the idea put forward by snowdrift.coop, encouraging people to set aside some money to fund beneficial open-source projects so that everyone can benefit. The more people pledge to support a given project, the more funding that project gets, growing exponentially. On the receiving end, it seems like a great way to get this kind of project funded, since the people - particularly big companies - each give a little money to build up the common goods in software.

On the other hand, it still relies on charitable giving. Yes, if you fund the development of, say, OpenSSL, which almost everyone uses, then you get active development and the benefits of a well-supported library with motivated developers and proper funding instead of a library casually (but passionately) developed by volunteers in their spare time. But if everyone else funds the project and you don't, you still get that benefit. I'm not clear how Snowdrift solves that problem, except that witholding your funding means a greater chance that the development will stall and you won't get the benefits at all.

Mokalus of Borg

PS - Which is just a statement of the snowdrift problem, I realise.
PPS - I'm not quite sure if that counts as circular.

Just get started

There are a lot of articles I've read lately about making games, specifically how to "get started". Most of them begin by saying "Just get started. Make something and you'll get better to make new things." That's not very helpful if the actual question was "I am thinking of making a game, and I don't yet know how to make images move on the screen. Can you help me?" and your response is "Just do it". There are some specific technical questions behind the first "how do I get started", and I think these game-making mentors should start with "these are the tools I use, this is specifically how I built my first game". That's more helpful than a vague motivational speech about keeping at your art to get the skills you will need later.

Mokalus of Borg

PS - Motivation is good, too.
PPS - It's just that specifics help more sometimes.

Blind computing

If you were designing a computer operating system from the ground up for blind people, what would that be like? You'd probably have to ask a blind person to help you, or blindfold yourself while you worked on it. Watching Tommy Edison using his Mac with the help of the screen narrator was like watching him fumble around in the dark. He was able to do it, but it was clearly blind-enabled as an afterthought to the primary design. Computers are about the one place where you'd think he would be able to find a level playing field, but all our normal desktop operating systems, software and websites are built assuming you have vision.

An operating system built for blind people would be - should be completely different to any existing desktop OS of today, just as a phone for blind people needs to be something better than stock Android on a featureless glass touch screen with an assistant to read out what you're doing.

Mokalus of Borg

PS - I've seen a blind man using a touch-screen phone on the train.
PPS - It did not seem easy for him.

Software shortcuts can make you a worse programmer

I read recently of some people lamenting the lack of skill shown by modern software developers and the difference between using tools to take over from already-mastered, mechanical processes in your work vs bypassing the learning of that branch of practice entirely. The argument was that it's perfectly fine to use tools to get around the parts of your job that you've already mastered, so that you can focus on more interesting, higher-level challenges, but a lot of modern software development provides tools that let you jump over that mastery from the start, which means you have no idea how it actually works. To make up an example, you might use a framework to read and write data from a database once you know how that works, because that's stock-standard boilerplate code you don't need to rewrite every time. However, if you use that code from the beginning, you'll never know how to connect to a database without it.

The problem is that you can't keep tools from people who don't know how to use them. You can't force a carpenter to use a hand saw until he understands it enough to graduate to a power saw. This goes doubly true in software. The tools don't care who uses them or what their skill level would be without the tool. The only thing you can do is challenge yourself. Start from scratch in everything and build up your own set of tools and frameworks over a career instead of picking up the biggest, baddest set of power tools from the start and demolishing a house by accident. That has to be self-enforced, though. If you couldn't have built it on your own, don't use it yet.

Mokalus of Borg

PS - You can't make people earn shortcuts, is my point.
PPS - Once they're open, they're open to everyone. That's kind of the point of software.

Jeeves is my new Belvedere

I've gotten kind of excited about a very simple program I've written recently, called Jeeves. It's a replacement for a program I got from Lifehacker that was called Belvedere (so I kept up the butler naming scheme). Basically, Belvedere is able to monitor folders on your hard drive, watching for certain conditions (such as files last modified before a certain date) and take actions in response, such as moving them to the recycle bin. I find it very useful for keeping my Downloads folder clean by setting it to watch for files older than a fortnight.

What bothered me was that Belvedere couldn't look for empty folders and remove those. I've tried various approaches to the problem, including writing a simple C# program to find and delete empty folders. That path started leading me to write a replacement for Belvedere, with configurable Trigger items and Actions to take in response. This turned out to be a very tricky and complicated way to operate, involving passing values back and forth in a generalised and highly flexible way, figuring out how to save and load rules and generally getting very messy. I abandoned the project for a long while.

Recently, I hit on the idea that maybe Python would be a better fit for the functionality, but I still couldn't figure out how to save and load the rules to be run by the generalised rule engine.

Then it hit me. Why do I need a generalised rule engine at all? If my goal is to run arbitrary actions in response to arbitrary conditions, then I should just write scripts to do what I want directly. Suddenly Jeeves, instead of being a complex, extensible file and folder monitoring rules engine, became a lean set of a few helper methods. Now my scripts are machine-specific simple Python programs that consist of statements like:

delete_folders(empty_folders("C:/temp"))

That's a Jeeves rule that deletes any empty folders under the temp folder. It's such an easy kind of rule for me to write that I hardly need to think about it. It also neatly sidesteps all of the problems I was having with the C# version. I schedule this file with the Windows Task Scheduler, and it works brilliantly. Belvedere, unfortunately, you're fired. I have Jeeves now.

Mokalus of Borg

PS - I recognise that this system isn't good for non-programmers.
PPS - Since it's just for me, that is not an issue.

Stagnant standards

One of the awkward things about technology is that we get the best use out of it when we have standards, such as common network protocols, but the most useful standards come out of ad-hoc solutions to emergent problems. So you need a way to send human-readable messages from one person to another, and we get email, but at first we get lots of different ways of sending email, which are incompatible with each other. Eventually we settle on one interoperable standard, and the world is good. Well, until Microsoft "embraces and extends" it, rendering themselves the keeper of the new ad-hoc standard. Ahem.

The other difficulty is that the standards we developed 10 years ago are now inextricably tied into absolutely everything, so even if they are no longer ideal or even vaguely appropriate, we have to keep using them because the status quo isn't going to stop or join you in a pre-emptive upgrade. HTTP 1.1 is probably the last version of that standard that will ever be produced, and the last of its kind as well. JavaScript may get some teeny-tiny upgrades, but it must maintain backwards compatibility with the websites of the 90s that it was designed to serve. Those had vastly different needs than today's interactive web applications.

So we get stuck in old standards, doing new things, and we will never be rid of them until the entire system collapses or someone tries something so fundamentally different that it demands a new ad-hoc solution.

Mokalus of Borg

PS - It's an odd pattern for advanced technology to take.
PPS - Or maybe it's a human pattern.