Ranting and Roaring

We’ve been blogging about this for the last two weeks, so I won’t bore you with all the details. You can get a copy here. Here’s what 1.5.1 contains:

• partial reworking of the UI for usability and clarity
• “universal searching” – try typing “Buffy the Vampire Slayer” in the search area, pressing the Search Button (the “Magnifying Glass” icon) and then double clicking on the red arrows under various search engines
• a roughed-in version of the BlogLines interface. Try File > Synchronization > Settings… to see what we’re doing

Here’s what’s missing, broken or needs further work:

• the search text field doesn’t deal with quotes, dots and what have you very well
• the search button should glow red when a search is taking place
• the search history button (between the text and the search button) doesn’t work
• searches cannot be converted to “persistent” – yet. This will be done by right-clicking on a search. This is part of our project to make “The Inevitability of Search Results as RSS” a reality.
• the BlogLines interface and code needs a lot more work. You’ll be seeing this in the next release.

You can configure Jäger to use a Proxy server by Settings > Jäger Preferences… and selecting the Network panel. If you have problems reaching certain sites, you may need to use this feature. You’ll have to get the HTTP Proxy information from your System Adminstrator or from your ISP.

I still haven’t been able to build a new Windows executable, despite the fact that I’ve downgraded wxPython to 2.5.1.5. Very depressing. I’m going to put a lot more effort into that today.

There’s an issue with my home computer where certain sites cannot be reached by HTTP: I need to go through a proxy server. I discovered this literally in the last half-hour. Solution: the next Beta version of Jäger will support Proxies.

I’m definitely going to support some of the BlogLines APIs (read more here). The first cut will:

• Allow you to enter your Bloglines account information
• Synchronize your Bloglines account to Jäger
• Optionally allow you to download new entries in blogs via Bloglines rather than directly through the Internet.

If you go to Space.com right now, you can catch a live webcast of SpaceShipOne‘s first half of its attempt to win the X Prize.

This could be very interesting. BlogLines is providing an API (press release; weblog link) that will allow desktop clients – i.e. programs like Jäger – to synchronize multiple copies on different machines; to provide notifications that a blog has been updated; and to provide truncated RSS 2 feeds that only contain new items.

I’m going to size this up this afternoon and decide whether this is worth implementing. I’m noticing a few shortcomings in the API, though some of these comments may be of nitpicking nature:

• There should be a way of marking items as read without re-downloading the feeds
• There should be a way of marking a range of items (possibly just by using two time points) as read – i.e. just because the aggregator has download some items doesn’t mean the user has read them on the aggregator.
• Why isn’t there a “download everything new” call? The current API requires that each subscription be downloaded as a separate HTTP call; why not have an aggregate function that returns everything in a single call?
• Why isn’t there an “upload blogroll” or “subscribe to” API (preferably the later) call? This is really required if desktop aggregators are going to synchronize with each other.
• Why aren’t Bloglines’ OPML extensions added as a separate XML namespace – i.e. use Bloglines:SubId=”3714388″ rather that BloglinesSubId=”3714388″?

Update: it should also be possible to get the OPML feed with only updated items too.

I’ve been very busy working on the next version of my news aggregator BlogMatrix Jäger, which hasn’t left me a lot of time for making blog postings or responding to emails, etc.. Don’t tune out yet, there’s a few…

I can’t create an executable version of Jäger due to some mismatch between py2exe and wxPython 2.5.2.8 (it worked with the previous version of wxPython). If anyone has a clue, please let me know:

 Traceback (most recent call last):   File "BlogJaeger.py", line 61, in ?   File "wxPython\__init__.pyc", line 10, in ?   File "wxPython\_wx.pyc", line 3, in ?   File "wxPython\_core.pyc", line 15, in ?   File "wxPython\wx.pyc", line 2, in ?   File "wxPython\wxc.pyc", line 9, in ?   File "wxPython\wxc.pyc", line 7, in __load ImportError: DLL load failed: The specified module could not be found. 

If this can’t be sorted out, I’m going to “downgrade” my version of wxPython and continue from there.

 -------------------------------------------- Qty      Item                            --------------------------------------------  1        The System of the World (The B            Sold by Amazon.com.ca, Inc. - taxed  1        The Pentagon's New Map: War an           Sold by Amazon.com.ca, Inc. - taxed  1        Star Wars Trilogy (Widescreen            Sold by Amazon.com.ca, Inc. - taxed  1        Futurama: V1                            Sold by Amazon.com.ca, Inc. - taxed  1        Norton SystemWorks Professiona           Sold by Amazon.com.ca, Inc. - taxed 

I’ve been very busy working on the next version of my news aggregator BlogMatrix Jäger, which hasn’t left me a lot of time for making blog postings or responding to emails, etc.. Don’t tune out yet, there’s a few things here that you may find interesting.

First, you won’t have to put up with this kind of trouble if you use Jäger (†Cosh). Sarah and Leo had built an RSS feed for CBC and CBC decided to shut them down, because we can’t have people reading the news. Jäger side steps this entire issue: it (effectively) builds you a private RSS feed for anything you want to read, including the CBC, the Toronto Star, or whatever.

The next version of Jäger is even more powerful, allowing you to search these sites continually for information you are interested in. Here’s an example of searching the BBC for news about the new Star Wars DVDs. The next version of Jäger is working its way toward being a generalized search engine to everything. I’ll have more news on this in the near future.

Finally, give it a try or give me a mention on your blog. Every bit helps right now! You know, that whole no job thing and so on and so forth.

I’m so pleased with the progress I’ve made so far with the next version of Jäger, I’m going to release the first beta tomorrow. It will show off a lot of the new integrated searching I’m working on include. For example, if you search for ‘star wars’ you will see in the search results special links for ‘Search: Amazon’, ‘Search: BBC’ all the way through ‘Search: Yahoo’. If you double click on the (red) folder arrow, a search for ‘star wars’ will be executed.

There’s lots of little usability factors to be worked out here but I think you’ll be quite impressed.

Here’s a screen shot of the next version of Jäger (so far):

You’ll notice a few changes already:

• Menu items that were previously in the Settings menu are now in a menu that pops up to the left of the particular major button.
• The lists are renamed slightly
• The icons on the main list are getting cleaned up
• The Search area is now at the top of the all the lists

I’m also working on deleting lots of obsolete code and functions that are too infrequently used to be worth keeping.

The Jäger Source Code is now available (and described) here. Have fun and if you have any questions, modifications you’d like to include, bug fixes or whatever – send me a note or leave a comment here.

This is my current thinking of what’s going to be in the next major update of Jäger, version 1.6. Not every change mentioned here will make the cut, but here’s hoping.

Simplified UI
There will be a small arrow button next to each of the major buttons (i.e. Favorite Weblogs, Recently Updated, and Search All) that controls the settings specific to that particular list.

There will be a “weblogs panel” that allow weblog specific settings to be modified across many weblogs at once. This will include whether a weblog is a “favorite”, is “hidden”, what Category the weblog belongs to and so forth.

Information relating to the offline view will be adjustable on the HTML page rather than within the UI itself.

Better OPML and blogrolling support
Our site “blogmatrix.com” will provide a central clearing house for your blogroll information. What does this mean in practice? It means you can maintain your blogroll in Jäger and it will be automatically updated on blogmatrix.com AND you can include this directly on your blog. No more editing your blogroll or having a blogroll out of date. An OPML copy of your blogroll will also be kept.

Note that Jäger already does many of these things but it’s fairly difficult to maintain. The intention is that should be almost trivial to use. It’s also optional, of course: you don’t have to export your blogroll (and if you do, you can select certain blogs to be “hidden”).

The interface to do this will be XML/RPC, so if you want to move the hosting elsewhere, you’re free to do that.

Better Synchronization
If you’re running multiple copies of Jäger, we’ll provide centralized synchronization through blogmatrix.com also, in much the same way we’ll do the blogrolls.

Improved Searching

We’re going to integrate the Universal Search Parser into Jäger. This means that when you type a search phrase, you’ll not only be searching the database but also Technorati, Amazon, CNN, BBC and so forth. Furthermore, you’ll be able to turn all these external searches into “persistent searches”: that is, you can continually check, say CNN, for a particular phrase and you’ll be notified when it appears.

If you don’t care for Jäger’s interface and prefer to use a different aggregator, you can still uses Jäger to set up persistent searches to feed your favorite interface.

Searching the Jäger database for blogs by name or by URL (currently done by placing “name:” or “url:” in front of the search string) will be done automatically.

GMail Integration
We’re going to attempt to integrate Jäger with your GMail account. That is, if you get new mail in GMail you’ll be notified in Jäger. This is still fairly experimental so we’ll have to wait and see how this works out.

You’d think I’d have a lot more time since my retirement. My old schedule involved starting work at 5 AM-ish; by 9 I’d have half a days work in. This ain’t happening so I’m behind on a lot of things I’d like to accomplish — high on the list, answering people’s e-mails. Please be patient with me for a little while.

BTW: Today is Trinity-Anne’s first day of school. She only goes for an hour the first day and I have to stay up there the whole time just in case she loses it. Not my little girl — she’ll be fighting me the whole way home to go back.

Here’s the Occam’s Razor FAQ about the CBS memos:

• Why are the memos fake? Because you can type exact replicas in Microsoft Word in five minutes.
• Why did CBS run the obviously (or obviously detectable) fake memos? They wanted the scoop (in particular, vs. USA Today)
• Why hasn’t – until now – CBS admitted they were fakes? Pride.
• Where did the memos come from? Never underestimate the guy in the tinfoil hat with lots of time on his hands telling people what they want to hear.
• Did the White House/Karl Rove drive this story († Cosh)? No. A million monkeys with typewriters will produce gibberish no matter how much time you give them; A million bloggers, some with degrees in Law and an eye for bull will produce at least Buckhead.

The following is most likely going to be the Jäger source code license. Your comments would be appreciated. This is a minor variant on the Sleepycat license; we’re going to insist that the pop-up contribute license stick around and that the initial “bootstrap” blogroll come from BlogMatrix.

Copyright (c) 2004 David P. Janes. All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
• Redistributions in any form must be accompanied by information on how to obtain complete source code for the DB software and any accompanying software that uses the DB software. The source code must either be included in the distribution or be available for no more than the cost of distribution plus a nominal fee, and must be freely redistributable under reasonable conditions. For an executable file, complete source code means the source code for all modules it contains. It does not include source code for modules or files that typically accompany the major components of the operating system on which the executable file runs.
• Redistributions in any form must not delete, modify or otherwise alter the code in “BlogWelcome.py” and/or “BlogManagerMixinSources.py”
• Redistributions in any form must not disable or otherwise circumvent calls to code in “BlogWelcome.py” and/or “BlogManagerMixinSources.py”

THIS SOFTWARE IS PROVIDED “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE DISCLAIMED. IN NO EVENT SHALL SLEEPYCAT SOFTWARE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

I’ve made some small updates to the USP:

• The example no longer needs the ‘text = ‘ parameter to search. It knows the right thing to do.
• I’ve added an ESPN seacher: ‘search:espn type:news buffalo bills flutie chargers pages:all’
• I fixed a bug in the scraper that was doing entity decoding within attribute values.

There’s a slight problem in my examples. You should use for now:

 import Search for result in Search.search(text = 'search:ebay TRS-80 pages:all'): 	pprint.pprint(result) 

The next version will handle the argument to search correctly (it’s possible to pass in a highly structured object that exactly describes the search).

Update: fixed.

Finally, my German lessons have paid off, in laughter.

Ivan + Jeanne = big eastern seaboard storm in 4 days?

What I’ve done so far, in no particular order:

• Got all my speakers hooked up to my 5.1 audiovisual centre
• Configured my amplifier; put it into THX mode for a laugh
• Tried to figure out how to acoustically isolate my DVD player from the room, as the house now vibrates when something exciting happens. Partial success.
• Watched Terminator 2, the special edition
• Watched Gladiator
• Watched half of Princess Mononoke; not impressed
• Raced sail boats, came 2nd.
• Released the Universal Search Parser, part of the Jäger source code.
• Rollerbladed
• Jogged
• Lifted Weights
• Iced right elbow

The support source code for Jäger is now available (licensed under the LGPL), http://jaeger.blogmatrix.com/source/. This encompasses three libraries, each in their own directory:

• generic: general helper classes and functions
• druecken: the Drücken HTML downloading and scraping libraries
• search: the Universal Search Parser

We will describe the files in “generic” and “druecken” at a later date. Some of this code is very useful, some less so.

The Universal Search Parser (USP) is an attempt to provide a consistent extensible method to utilize online search resources within a Python program (and in the future, as RSS or RDF results). It’s quite easy to use; for example, here’s how you can get all the listings for TRS-80s available on EBay:

 import Search for result in Search.search('search:ebay TRS-80 pages:all'): 	pprint.pprint(result) 

Yielding:

 {'Bidders': 0,  'BuyItNow': u'$15.00',  'Price': u'$10.00',  '_link': 'http://cgi.ebay.com/ws/...1247&item=5122154379&rd=1',  '_title': u'Sol 20, Exidy, TRS-80, NorthStar, BASIC, Assembler, etc'} {'Bidders': 4,  'Price': u'$15.50',  '_link': 'http://cgi.ebay.com/ws/...=74947&item=5123221664&rd=1',  '_title': u'RADIO SHACK TRS-80 POCKET COMPUTER MODEL PC-4 W/CASE'} {'Bidders': 0,  'Price': u'$0.99',  '_link': 'http://cgi.ebay.com/ws/...=74947&item=5123225351&rd=1',  '_title': u'Downland for TRS 80 Color Computer'} ... 

What else can we search? Right now, we have the following modules implemented (there are example searches within each file):

• SearchAmazon.py (requires pyamazon)
• SearchBBC.py
• SearchCBC.py
• SearchCNN.py
• SearchCanada411.py
• SearchEbay.py
• SearchGoogle.py
• SearchIMDB.py
• SearchSourceforge.py
• SearchTechnorati.py (requires pytechnorati)
• SearchWhitepages.py
• SearchYahoo.py

To install the USP (and all the other libraries), do the following:

• download the latest copy of “jaeger-support-*.tar.gz” file
• unpack it
• “cd search”
• try “python Search.py ‘some query‘”

The main source code for Jäger will be along later this week or by Monday at the latest. The USP will be integrated into Jäger for both “immediate” and “persistent” searching in the very near future.

If you give this code a try and you like it (or hate it), please send along an e-mail (or even send a donation): I’d love to hear from you in any case. If there’s any other Search modules you’d like implemented, send me a note and I’ll see what I can do, or if you’re so inclined try it yourself – it’s not too difficult.

The rumors are true. F1 Racing is reporting:

The Renault F1 team announced this evening that Jacques Villeneuve will test for the team at Silverstone on Wednesday and Thursday. If eveything goes according to plan Villeneuve will also replace Jarno Trulli at the Chinese Grand Prix next week.

[...] “We have reached an agreement for Jacques to test for the team at Silverstone on Wednesday and Thursday,” team chief Flavio Briatore confirmed. “This test might lead to Jacques joining the team for the final three races of the season beginning next week at the Chinese Grand Prix.”

Update: See the comments! He’s gone to Sauber.

Can New Orleans take a slow moving Category 3 hurricane?