In case you wondered ...
… why I haven’t blogged much recently, and what happened to Blunt, no worries. I was (and am still) working on something else for my trusted Newt :) I should have something ready to be released soon, and hope that people will find it useful!
The Warnings...
… in the header file used to interface with the serial chip seem to be quite appropriate. It is not really rocket science to send and receive data, but it is pretty difficult to pinpoint any problems because they are usually timing related. I made some changes in the sending part which enabled more stable OBEX operations with Blunt 2, but it looks still a bit incomplete - sending data in large batches results in errors on the receiving end, which hints at some data corruption.
Getting closer...
I just used Blunt 2 for first time together with Neo and IC/VC, successfully transferring data from the Newton to a PC over OBEX. It seems that there are only two things open to enable basic functionality: instability caused by unclear memory management when handling Newton events, and flow control for sending and receiving. Neither is impossible to resolve :)
Overall, this was a quite important milestone!
Messy
I’m still trying to solve the flow control problem with Blunt 2. I’m using a Linux PC to send data at the highest possible speed over RFCOMM, and without any handling besides the raw BT stack, it sort of works, but it is buggy and crashes the Newton. The code which is necessary to make it stop crashing, as well as the code needed on the NewtonScript side to actually do something useful slow things down, causing buffer overflows. The flow control code which I put in didn’t really work, but I’ll still see if the more advanced credit based control would work better.
On the bright side, it seems that when using MacOS X, the speed is just about right for things to flow smoothly - about 5kByte/sec, including handling on the NewtonScript side (I’m storing the data in a 1k binary object). That means that I only need to add the sending part, and the scenario where the Newton initiates the connection should be complete - sending and receiving should work, with much improved error recovery. No more -18000 and -19000 errors :)
10 to 15 ...
… is the raw transfer speed in kilobytes per second which Blunt 2 manages. This is however not taking any overhead into account which is still missing, most notably sending the data from the Blunt stack to the CommTool, and then using that data in NewtonScript. It’s very hard to guess how much the overhead actually is, but the above number is quite good. I also managed to solve some stability issues with the stack, the only two remaining stability issues are the potential paging out of the interrupt handler code, and the implementation of the CommTool.
Flow control...
… seems to be something I need to put in place next in Blunt 2. Just for fun, I wanted to test the raw transfer speed over Bluetooth, and was spending quite a lot of time trying to figure out why I get corrupted data after a certain time. Turns out that the receive buffers on the PICO card are too small, and with all the logging, Blunt 2 is reading too slowly from the card, resulting in an overflow. The good thing is that with timers being now much easier to implement in Blunt 2, adding flow control should be not too hard. Ideally, this would be credit based Bluetooth 1.1 flow control, but it looks like that the Mac OS X stack only does the serial port emulation of the 1.0B version, which is a little less elegant.
Testing, 1, 2, 3, Testing, ...
Most of the pieces in Blunt 2 for sending data from the Newton seem to be in place - I can successfully discover devices, pair, get services and connect using a regular NewtonScript endpoint. This last part is actually pretty important. There are however still some quirks, the most annoying issue is stability. I think that both the server and the CommTool have problems in this area. The server probably because it is using the TSerialChip directly, and there are lots of funny things going on with DMA and interrupts, and the CommTool looks problematic because it is tied very closely to the NewtonScript world and its event loop.
Still, I think this will not be magic to sort out, and in the meantime, I’ll try to get an idea what the performance of Blunt 2 will look like. One promising thing is that the always on and multiple simultaneous connection features look feasible. And that opens the door to fancy things like browsing your Newton from a desktop PC via OBEX, mapping for example Names to vCard “files” which can simply be copied, or Notes to “files” with multiple representations (e.g. “Note.nwt”, “Note.xml”, “Note.txt”).
PS: Native Bluetooth support in Windows XP is pathetic, third party software like Widcomm is unstable and Mac OS X is so much better - there, I just had to say that after a day of fighting with Windows ;)
Wiki
I just moved to MediaWiki to maintain 40hz.org, it makes updates much simpler - hopefully that will encourage me to update information more frequently :) There is still one issue regarding uploading packages, source code and other data (the Wiki is not really made for that), one option is to use SourceForge exclusively as the data repository. That would have the advantage of having only one location for multiple releases, but the disadvantage is that it is a bit more complicated to download things. Also, direct download of packages would not work.
I will gradually move most of the pages on 40hz.org to the Wiki, but for now, pages besides the main page are still “regular” pages. I also disabled editing by non-registered users, and since I’m the only registered user, that make things simple :) But it might make sense to allow others to edit pages in the Wiki, let me know if you have specific ideas. There is of course always the WikiWikiNewt on UNNA.
Blunt 2 Status
In case people are interested, here’s where Blunt 2 is standing at the moment: Bluetooth Setup looks quite ok, but getting Service information from remote devices is not implemented. Im testing with my Pico card, which hopefully helps people who are using the same card (it’s a cheap and readily available choice). I sent a version earlier to Daniel Padilla, unfortunately it just crashed his machine. My current development machine has been brain wiped, so it should be in a state similar to other Newtons, let’s see if I can prepare another version for testing. Ideally, people who want to test have Hammer installed so that we get a bit of logging data.
There are some open issues regarding the initialization of the components of the Bluetooth stack, i.e. how the HCI, L2CAP and RFCOMM handlers are started up. It’s a bit hackish at the moment, starting from the bottom of the stack and moving upwards. Maybe initializing the high level layers first is cleaner.
Brain Transplant
One problem I have sine I bought my first Newton back in 1995 is that my handwriting is bad, in fact I have trouble reading it myself ;) However, Newton’s printed recognizer is actually doing a quite good job. To improve recognition, I use alt.rec…, and since most of the things I write in English, I finally decided to do a brain transplant, moving the US ROM from one of my spare Newtons to my main Newton. The physical process is very simple, but moving data back and forth required a bit more planning. I wanted to take advantage of the fact that the memory gets wiped completely after a ROM transplant and decided to see if I can preserve all my data without a backup. For that to work properly, I moved all packages to my memory card, and then used Soup Mover to move the data as well. Settings are lost, but they are usually not that crucial. Everything worked fine in the end :)
Connecting - Disconnecting - Connecting ...
The overall design and implementation of the Blunt server seems to work ok, I can connect from other devices for pairing and getting Bluetooth services, repeatedly opening and closing connections. This is actually the biggest difference to the current Blunt implementation. Now it should be possible to have services running permanently, and not only when a component uses Bluetooth. Bluetooth Setup is also now starting to get more and more adapted to the Blunt server, which makes discovery problems hopefully a think of the past (the discovery time will be configurable as well). One important thing to sort out still is memory consumption, I think I need to add malloc/free tracing macros to the code.
Down to the metal
Another solution to the GCC problems for Newton development is to use Assembler. Sounds crazy? Maybe, but that’s how I started programming ages ago, and at least for low level stuff like Blunt (which is mostly about moving bytes around), it might actually be quite suitable. Only interfacing with the NewtonOS and C++ requires some effort, but there’s no reason not to use clever preprocessing or macros to simplify it… this is of course no cure to port existing C or C++ code.
GCC again
Now that the new Intel Mac mini is out, it would be nice to upgrade to it at some point, but the question of the old Newton development tools comes up again… I guess there is really no way around replacing the tools with GCC and NEWT/0. For GCC, the problem is that a new backend is needed to take care of the proper creation of the vtables, and last time I looked, NEWT/0 does not create NSOF output, but rather NewtonScript bytecode for functions. Well, I guess neither problem requires rocket science to be solved, just some work :)
Putting things together
It looks like the architecture of Blunt 2 is now solid enough to bring in the missing functionality one piece at a time. Currently, my goal is to adapt Bluetooth Setup to use Blunt 2, and at least the lower layers are already in place. Driver selection, resetting the module and discovery are working. Lots of multitasking happening, and it’s nice to be able to use more and more features of the actual NewtonOS like message sending and tasks!
Next Action...
I’ve been doodling around with implementing some stuff from “Getting Things Done” on my Newton, and it seems to work reasonably well (I was getting too annoyed by abusing my Inbox as a to-do list and working on stuff as it comes up instead of working on stuff that’s actually important). I’ll probably write a bit more about this later, right now, the biggest problem seems to be my handwriting ;). I implemented some stationery, and I’ll probably add some more hacks to the Notes application, as well as integrate a bit better with DateMan. All that will go of course to 40hz.org when it’s ready. And then, I’m going to get back to Blunt 2, promise!