We've been pretty busy (and quiet!) recently, making sure that the first release of Brandwatch is a big success, and on time. It has been exciting: seeing this ingenious piece of software coming together, bit by bit, feature by feature, and becoming a fully-functional whole, meeting our expectations. All of this thanks to a great team of developers, who consistenly delivered on the requirements, and dare I say some decent management too!

Seeing Brandwatch at work has definitely been the highlight of the last few months. We've been spotting stories as they emerge on the net, from the opposition to bank penalty charges, to a poisoned dog food scare in the US, via rumours of a hard-drive being launched for the Wii (and yes, the PS3 is a bit of a bomb).

So the obvious question - what's next?

We're going to spend our Brighton summer not on the beach, but busy - still - delivering more of the same, and more of the different too. We're going to analyse more industries, more brands, and track how brand discussions break down into topics, evolving and interacting over time. And always refine the engine scalability, the brand matching, the intelligent sentiment analysis. Finally, Brandwatch will open up to users, letting them interact and build on the information, making it an even better application.

But for now, let's enjoy the birth of Brandwatch.

Dr Fabrice Retkowsky
Technical Director

posted on: Thu, 07 Jun 2007 | permanent link to this entry

As Giles pointed out in his posting, "The Future of Search", the enemy of accurate search is quantity. With huge amounts of information being added to the web daily this is something that both the suppliers and end users of web search should be worried about. The more information out there, the harder it becomes to classify, qualify and rank. This means users wading through a lot of hits, or refining their search, and when it takes too long it stops being useful. These days search engines are not the only way of finding answers from the web. An argument is as likely to be settled by a search on Wikipedia (a library-style solution) as one on Google.

So what does this mean? Is search, as we know it, going to be overwhelmed? Could we see the renaissance of the library, the newspaper, the record shop - because we trust our librarians, editors and that music-geek who can name the artist, album and drummer's rehab clinic from two lines of song lyric that I barely remember? Probably not (he's been a little too snooty ever since I falsettoed 'where the grass is green and the girls are pretty'), but it is possible that we may see a move away from monolithic single search solutions. Contemporary search just isn't making the necessary value judgements. Search algorithms can't pick up on all the non-verbal communication or draw upon the healthy wad of prejudice that allow the music-geek to place my interests in the genre of post-punk ex-grunge ex-vegetarian thirty-something Guardian-reader. Online, the context of a search is lost - the search engine doesn't know that I'm standing in my 'Drugs Kill' Sex Pistols t-shirt in a fringe record shop doing its best to survive in post-iPod Brighton.

Step up personalised search. When Amazon tells you that someone else who bought The Anarchist Cookbook also bought Delia's Complete Illustrated Cookery Course, that's a personalised search even if we don't immediately recognise it as one. The concept is simple. It uses your order history and the order history of every other customer to try and second-guess what you might want to buy. This is interesting because it's both passive and persistent. The user does nothing, except maybe buy new goods; Amazon continually checks through its order history to look for other people who ordered the same things. When I get offered a new item it's either because I bought something new or someone else bought something I already have. More interesting is that this is a human-powered search. Amazon have effectively enlisted all of their customers as unpaid researchers. No one is having to tune algorithms that try to predict the nuances of purchasing habits, they're just exploiting the huge amount of statistics that they have gathered to make some educated guesses; the end result is all the more interesting because of it. Amazon knows that people who buy a CD by garage band X are likely to also want to buy one from garage band Y not because their computers know anything about the garage music subculture but because their customers do.

Of course there are other human-powered search systems, and I don't mean the ones where a professional researcher is paid to do your searching for you. The so-called 'folksonomies' used by del.icio.us and flickr are the same thing. These systems better the paid researcher because not only is the responsibility distributed over thousands of users, increasing the accuracy, but they have a sense of timeliness; tags rise and fall on the ebb and flow of the zeitgeist.

Personalisation and folksonomies both work because they see the web as more than just inter-related content. They recognise that the users are as important as the data and that by interpreting user interaction with that data, and perhaps with each other, they can improve relevancy. Thinking about how search applications can best harness these new models, and how we might come up with new models is what makes working in search interesting.

Matt Roberts

posted on: Tue, 07 Nov 2006 | permanent link to this entry

They warned me, but I didn't believe them. You won't be able to stop, they said - the boundaries will blur, and you'll be working all hours. Ridiculous, I thought - when I'm working from home, surely the problem is going to be doing any work whatsoever. Especially as I'm moving to Paris to do it; the city of love, fine bakery products and extravagantly foul-smelling cheese is certain to lure me away from my desk-cum-kitchen-table.

But I find myself here, hunched at what can only be described as an anti-ergonomic workstation, nearly two weeks after the new home-working regime began - and evening is creeping into night once again. With none of the normal signals to send me home at the end of the day, as well as the literal impossibility of going home when you're actually already there, the urge to just get that last! little! bug! fixed! is proving irresistible. My partner is threatening to get me a "But it's almost finished!" tattoo, and worst of all, I haven't seen a single second of French daytime television.

Part of this inability to stop is the wonderful lack of distractions. Magpie's office in Brighton is generally a good, quiet place to work - but once you get a certain number of people in a room at one time, some noise is inevitable. I'm a developer, and developers really don't like to be disturbed when working - tackling a gnarly coding problem in a busy office can feel like building a four-deck house of cards in a wind tunnel.

But putting aside my stereotypical coder's hat for a moment (and yes, it does have "got root?" on the front, since you ask), the modern remote worker is spoiled for choice when communication with his or her less-fortunate office-bound brethren becomes necessary. Email, instant-messaging and IRC are already the methods of choice for many of my colleagues - even between people on opposite sides of the same desk. Almost all communication with my project manager is over IM; more immediate than email, yet less synchronous than a phone call, it's the perfect way for both parties to talk at their own pace. And when a phone call really is necessary, I can dial the UK with Skype for one cent a minute, or have UK clients dial a London phone number which routes through to my Skype phone.

For me, communicating using IM and email is much easier with people I've met face-to-face already. Not that it's impossible to collaborate remotely with people you haven't met, or I wouldn't be writing this in OpenOffice on GNU/Linux - but text-based communications with people you don't know very well is much more prone to ambiguity and misunderstanding. Luckily, having been at Magpie for six years, I know the people I'm chatting with well enough to be able to gauge just how much more I can goad them about their intranet photograph before they buy a Eurostar ticket to come and break my knee-caps in non-electronic person.

Magpie, being a tech company that's had remote workers since its inception, has the necessary technical infrastructure in place to support them. As well as ensuring a documented audit trail, project mailing lists and ticket trackers are great for keeping us "remotes" (I'm hoping it will catch on) in the loop. Code is kept under version control, facilitating safe collaboration. And as I can connect the Magpie network from anywhere over a secure SSH connection, there's very little I can do in the office that I can't do from here. (Though hiding in the cardboard box the server came in and jumping out when everyone gets back from lunch is considerably less fun on your own, I've discovered.)

Going to meetings is the obvious exception. Remote workers can "attend" by conference call, or even by video conference, though I've yet to experience one of those from the far end - for now, I'm safe to code in my new Yves Saint Laurent pyjamas without anyone being the wiser. Conference calls have worked well for me so far, although I'm appreciative of the benefits a web-cam can bring; when you're represented by a wall-sized projection of your face in terrifying close-up, you instantly gain some kind of primal upper-hand.

I'm learning the success in teleworking lies in starting when you should, working your hours, and then stopping. The temptation is to mix-and-match work and play, because you can - but that way lies madness, unhealthy sleep patterns, and confusion for your co-workers who never know when they can get hold of you. When I stick to the pattern, there's really very little difference between working here and working at my desk in Brighton.

The croissants are better here, though.

Joe Holmberg

posted on: Thu, 12 Oct 2006 | permanent link to this entry

Ten years ago, we didn't know what a Search Engine was.

Alta Vista was the first one I can remember and it held the limelight for a short period back in 1997. Then along came Google. At first it was a kind of underground thing, then suddenly, bang, the whole world clocked on. Google ads appeared and now we have a $100bn super company. Now I find it hard to imagine how we would manage without the search engines.

Lots of things came together to cause this search phenomenon. The internet, free information and the fact that most webpages have very little structure. We like using search engines as they are very simple and very forgiving. Not structured like a complex file system or database. I have learned to hate the file structure that I have devised for myself on my home pc. I can't remember where I put the file I want. Why isn't it in /my documents/clients/client_a/project_b/project_plans/internal folder? Basically because my memory isn't as good as my computer's. I thought that creating a complex file system would help and sometimes it does, but mostly it doesn't. Enter free text search. No file system. No database, just a searchbox. It's so easy. And when it works it's such a relief. No more delving into directories within directories looking for some clue as to where you saved the file.

But the enemy of free text search has arrived. Quantity.

Data is being produced at an ever-increasing rate. Google have stopped publishing how many pages they index, I'd guess it's more than 20 billion. There are more than 30 million bloggers around the world. Finding what you want using free text search is getting harder, and yet we don't want to go back to the nightmare file system which takes ages to use and is so frustrating. So what do we do?

As an MD of a Business search company, I have wrestled with this question. The answers I have so far fall into two categories:

1. Try to limit the amount of information going in
2. Get to know the person using the system and structure the information getting to them based on who they are and what they want

There are of course dangers with these approaches. If you limit the amount of information going into the system, there is a chance that you will miss some of the most important stuff. So you need experts to figure out what is important and a system which can be trained.

Getting to know the people using the system isn't easy either. The basic principle is that when someone clicks on a search result, you can learn something about them and potentially when you serve them up with their next set of results, you take what you have learned and you feed that into the system. An example of something similar that works amazingly well is the Amazon “Customers who bought this item also bought...” But of course my motivation for buying a particular book may well be different to yours so the suggestion could be redundant. There is no doubt though as Chris Anderson explained in his piece “The Long Tail” http://www.wired.com/wired/archive/12.10/tail.html back in October 2004 that if you get this kind of information mining technique right, the results can be staggering. We at Magpie are using these two principles in the development of our suite of search software. In particular Search & Alert Pro which contains professional information libraries that are designed to meet the needs of specialist information professionals. Have a look at what we do, sign up for a free trial, see what you think.

We are at the beginning of the journey into personal information delivery. There are lots of big forces colliding – the owners of the information, the delivery channels like the big search engines, the advertising revenues, and also popular opinion and the social web all have different objectives. 2006 is the year we started finding it hard to find things again. 2007 will see the launch of pioneering products like ours, which will approach this problem head on, and help us get back in control of the information. Exciting times are ahead.

Giles Palmer

giles dot palmer at magpie dot net

posted on: Tue, 12 Sep 2006 | permanent link to this entry