What’s next
Ok, the most obvious things to work on next would probably be the bugs and feature requests from Sourceforge. One thing which is however not there is to increase stability for Blunt. This covers two things: proper shutdown of the driver in error or disconnect cases to avoid the dreaded “communications port busy” error. And then safeguards to prevent crashes when talking to devices which I cannot test with. This concerns mostly Sony Ericsson devices which is actually a bit surprising. Ericsson has always been a good Bluetooth implementor, so I wonder if they interpret the standard a bit differently than I do. One thing they certainly can do is to be more agressive in their implementation since they know what should work.
Surfin'
I now have a PPP daemon running on my Cube, accepting incoming connections via Bluetooth. Works like a charm for mail and web surfing with my Newton, using Blunt. It’s almost like WiFi, only the power consumption is lower. The setup is quite simple: Add a new serial port on the Mac via the Bluetooth Serial Utility, add a Bluetooth internet setup with the correct name server and other parameters, enable IP forwarding with ipfw and natd and start pppd on the new serial port.
These are the commands I used to enable IP forwarding (en1 is my outgoing interface):
/usr/sbin/natd -dynamic -interface en1
/sbin/ipfw -f flush
/sbin/ipfw add divert natd all from any to any via en1
/sbin/ipfw add pass all from any to any
/sbin/route add default 10.0.1.1
I’m behind a firewall, so there are no other fancy filtering rules. The firewall is at 10.0.1.1, so that should be the default route to the outside world. The pppd command line looks like this (Bluetooth-Serial is the name I gave to my serial port):
/usr/sbin/pppd /dev/tty.Bluetooth-Serial 230400 noauth local passive proxyarp asyncmap 0 silent persist :192.168.1.110
When discovering services from the Newton, there are now two serial ports, and one of them is the new serial port. The other is the PDA sync port. Unfortunately, the exact port numbers seem to change, but it’s easy to find out using trial and error.
Bluntness
The latest changes to Blunt made the L2CAP connection establishment phase a bit more stable. It’s not really clean, but works: Instead of using a timeout and retransmit in case the connection establishment stalls, I just send the packets twice, no matter what. They’re marked using the same packet identifier so the peer can throw the second one away. Likewise, I ignore any duplicate response. A more difficult issue is connection closedown. It seems that proper disconnect at least in the RFCOMM layer is needed. Otherwise, peer devices don’t finish an OBEX transfer (or discard the transfer such as MacOS X). It’s a bit trickier to get into closedown since I’m not sure how much of the comm tool is still usable at that point. First attempts just caused a reboot.
Still open is also the SDP and some nasty crashes using the P800. I’m not sure why the P800 causes the crashes, but I’ll probably have to add some safeguards. The SDP is likely easier to fix. Some improvements for Windows support are also needed in that area.
Unrelated to that, adding gzip support to Courier was certainly a success. 8k/sec over GPRS are quite nice. Still, NHttpLib could do well with a rewrite…
Clock offset?
One theory for the missing Bluetooth packets on connection establishment is that I don’t send the clock offset when connecting. It doesn’t seem to matter on the 3650, but maybe the USB dongle on the Mac works better if the clock offset is available. It helps keeping packet transmission in sync… for now, connections to the USB dongle succeed only the second time around, and my guess is that the clock offset is at that point known.
NIE Support
After exploring the various bits and pieces which NIE installs, I managed to put together a first version of a “NIE Nitro & Blunt” module. It reuses much of the Serial & Modem module and adds a new internet setup (IrDA for now, Bluetooth will follow). This will make the patched NIE module (which has to be currently used with Nitro) obsolete. One open issue is the connection slip for Blunt since that requires a field for the peer device, but I don’t expect that to be too complicated. Getting closer to true Bluetooth integration…
More Bluetooth Setup
Discovery and Pairing look ok. There is still the issue with crashes after sending data (or a -48200 error), and Neo should not register transports for IrDA and Bluetooth when they are not installed. But the Bluetooth routing slip now uses the discovered device information and sets the pairing link key if available. The current archive on the Experiments page contains these latest developments.
The tricky part is now to get the same level of integration for Internet Setup… and not to forget about a more intelligent SDP server implementation.
Back to SDP
I made some progress with the Bluetooth Setup application (it’s part of the Blunt.sit archive on the Experiments page). Discovery works, but overall, it’s still quite rough. The preferences for chosing the Bluetooth device settings are integrated as well and are fetched by the comm tool when an endpoint is created.
The next steps would be pairing and discovering remote services. This probably one process, i.e. pairing also gets the available services on the peer device. This requires some additions to the SDP layer of Blunt. Not my favorite part…
The last steps are to use the discovery and SDP information in Neo to build the route slip when sending data to another device. I also have a vague idea on how to do the same for Internet Setup, making it possible to chose a peer device for a PPP connection. If my idea works, it will help Nitro as well and make the patched NIE Serial module obsolete.
Much better
The Blunt PPP speed problem was indeed caused by the wrong MTU settings. While trying to fix a problem when sending data via OBEX, I saw that the code for correct data segmentation during sending was still missing. The workaround at the moment is to use a small MTU size which will fit into the HCI buffer to the card and maybe add segmentation later. PPP over GPRS works fine now… the Newton is not a speed daemon, but it is definitely enough for browsing the web with Courier. And there is still the option to add gzip compression to NHttpLib.
Tricky, tricky…
The interactions between the various buffers in the TAsyncSerTool seem to be clear now. I reverse engineered the EmptyInputBuffer method more thoroughly and found out that the comm tool tries to get as much data into the NewtonScript buffers. There is still something to do about data which won’t fit… it looks like the async tool will lose some of that data because it might be overwritten. Or maybe it will use flow control. Anyway, I’ll just store it in an intermediate buffer which is hopefully big enough for all higher level protocols. For OBEX, it is 2k, and PPP should be happy with 1.5k. Having this now out of the way I can continue working on the Bluetooth setup application.
Now where did that byte go?
I’m currently working on the Bluetooth setup application, or to be more precise, on the device discovery feature. The Bluetooth setup application will be used to discover devices, pair with them and set system-wide parameters such as connection speeds.
In the process, I hit one of the tougher problems in projects like this: Figuring out what the existing NewtonOS code expects to happen in a new component (such as a new comm tool). It’s a very tedious process… currently, there seems to be an off-by-one error in the data transfer back to the NewtonScript world. For certain input scripts (mostly those which read byte by byte), the last byte in a transmission is lost. This bug also causes the PPP negotiation phase to take forever.
I also still haven’t found out why PPP is so slow. It seems that the peer device simply takes its time talking to the Newton. At least on the PPP and HCI level, I can’t see any retransmits or packet errors. But I’m working on it.
Rough Edges
With Courier 0.5 out, I’ve got now again some more patience for working on Blunt. Caching and download were something that just had to be in Courier, but now it’s Hammer time again.
One remarkable step forward today was a successful PPP connection over Bluetooth to my Linux machine. There is however something seriously wrong with either the MTU size, the ppp daemon or flow control: Transfer speed is about 400 bytes per second. Not really breathtaking. That prompted me to get OBEX working reliably to get some alternative numbers. There were (and probably are) still some rough edges in the L2CAP and RFCOMM layer. A remaining mystery is why some packets get garbled right after connection establishment. But simply sending them again helps.
Anyway, the speed over OBEX is more satisfactory: about 10-15k/sec from my Mac, which is quite good considering that the serial speed to the Bluetooth card is 23k/sec. I guess that the slow PPP speed is related to the failure of establishing a PPP connection to my 3650…
LPR Driver
There was an interesting discussion on NewtonTalk on printing. I do not print very much anymore - it was exciting in the beginning, with devices such as the toilet roll printer that worked with the ZX Spectrum, or the first inkjet printers - but the amount of material I would have to print for doing my work is simply too high.
But on the other hand, a challenge is a challenge, no matter how you look at it! So I investigated how much it would take to enable TCP/IP (or rather, lpr) printing from the Newton. And it seems quite feasible. Praise the NewtonOS again for that. A lpr printer driver would be a subclass of the TPSPrinterDriver protocol class, similar to the TPSPAPPrinter which does AppleTalk printing. I played around a bit, and got a dummy printer class working. It only needs the code to initiate the connection and send some data. Unfortunately, connecting over TCP/IP requires grabbing a link and doing a DNS lookup, something which wasn’t really planned in the current printing process. I guess it would have to be done via a proto patch of the printing transport…
Caching
I’ve been fiddeling around with Courier over the last couple of days. One of the planned improvements is caching of pages in the history and it seems to work reasonably well now. One problem however resurfaced: The protoTXView can use a VBO to store large text files, but it seems to be somewhat unstable. Sometimes, the stored data gets corrupted, resulting in -10600 and -10061 errors. The workaround for now is to have either history caching or VBO usage.
“Let things settle”
Sometimes, things just don’t want to work as designed, you encounter Heisenbugs, random errors show up, and finding a rational explanation is way too much work. In these cases, “let things settle” is a perfectly valid strategy! L2CAP connection establishment was a bit flakey in Blunt, so I added a command to get the link quality before initiating the connection, delaying things a bit. Seems to work so far. And I got the first bytes sent over from the Newton as a connecting device to the Mac machine.
Starting a PPP daemon on the Mac to let the Newton establish a TCP/IP connection via Bluetooth should be possible now… but there is still lots of additional code required, especially the Internet Setup/Blunt coordination is missing (stuff like peer discovery, finding the correct RFCOMM port, handling pairing etc.)
Pieces of the Puzzle
Connecting to a Linux machine via Bluetooth works now. Data transfer seems to work as well, but that’s not really verified. The 3650 and Mac OS X don’t like some of the connection parameters used in the top level RFCOMM layer, but much of the lower layers is implemented. Some mysteries still need to be solved, for example the timing when sending packets via the HCI layer. Sometimes, the packets get lost… what would help is using delay timers, but that’s an area I’ve tried to avoid so far. Timer events should however be a universal feature on the NewtonOS, and they would help in making the Bluetooth connection more reliable.
I also cracked open a no-name USB Bluetooth dongle to see what kind of module is inside. Unfortunately, Bluetooth modules often come in either UART or USB flavor, and the module I found is a USB only module. So much for getting cheap internal Bluetooth on the Newton! The modules from BTDesigner.com are not that expensive, but shipping is about $30…
One thing that got a bit in the way of my development work is that Mac OS X 10.2.8 broke my Bluetooth setup. The system preferences crash in the Bluetooth panel (yes, I’ve deleted all relevant .plist files) and my Acer dongle doesn’t work anymore. Seems like Apple wants people to upgrade to Panther.
Buffer Sizes and intriguing thoughts
I’m almost ready to release a proof-of-concept version of Blunt. It still crashes occasionally and sometimes, a connection can’t be established from Mac OS X. But otherwise, I’ve been able to successfully send files from my 3650 and desktop to the Newton. The bug du jour was a buffer overflow, causing some bytes to be lost here and there, but that’s fixed now. The buffer sizes in the various places in the stack definitely need some adjusting!
An interesting related topic are of course Bluetooth cards. The PICO card which I’m using is neither very battery-friendly nor small, so a nice, small, low-power alternative would be great. And it doesn’t have to be a PCMCIA or CF card: It should be possible to put together a small PCB with a Bluetooth module and fit it inside the Newton, similar to the SER-001. Bluetooth modules are tiny, all-in-one solutions which can be interfaced with using serial lines and which don’t need very much extra circuitry…
Wow!
Wrestling with buffer management between NewtonScript and Blunt … getting hit by flow control problems … wondering where all the bytes went … crashing and crashing and crashing …
… and then a quick, stupid test: Why not let Neo instantiate an RFCOMM endpoint instead of an IrDA endpoint? Change one line, add some instantiation options, initiate receiving on the Newton, open Bluetooth File Exchange on the Mac, hit send … it works!
Now the first file to be ever transferred over Bluetooth from a Mac to a Newton sits in the Newton’s inbox. Quite neat. I guess that’s worth a nice Westmalle Dubbel.
Seven Bytes
… made it safely from the Mac OS X Bluetooth File Exchange application through various layers of protocol handling code in Blunt to show up in a text view on the Newton. The seven bytes are an OBEX connection request.
Now I’ve got to add code for sending data back, check the necessary flow control mechanism and the first third of the Bluetooth integration is done. It would already allow sending files to the Newton via Bluetooth. The second third is initiating connections instead of listening and the last third is cleaning up things (mostly SDP) and adding a setup application.
Shudder!
Who did this!? It’s like talking in gzip! … what I’m talking about is the Service Discovery Protocol (SDP), used to find the services running on top of a Bluetooth stack. It uses a very compact binary data representation which might be space efficient but very inconvenient to parse. Oh well.
On the brighter side of things, I’ve managed to restructure some of the higher level Blunt code without breaking too much. Now there is some degree of multiplexing possible. I also had the idea of using a permanently operating Bluetooth stack which could have multiple comm tools as clients. This could be really interesting to run multiple services on the Newton as well as sending data at the same time. I have to remember why that idea didn’t work with IrDA, there must have been a catch.
Totally unrelated, but in case anybody missed it (that’s not really possible, is it - there is this really practical RSS channel on the 40Hz SourceForgepage !): Courier 0.3 and Raissa 1.1 are out.
“Pairing was successful.”
From a Newton to Mac OS X and a Nokia 3650, using Blunt. Not too bad. At the moment, the code looks a bit hacked together (well, that’s what it is), and the usual “hack - refactor - hack” cycle has to start sometime. But for now, it’s good enough. Next up is initiating connections.
Steady Progress
At the moment, it’s fill-in-the-blanks for Blunt. The Bluetooth specification is quite clear, I have two peer devices to test with, the comm tool integration is working so far - now all those functions, states and packet formats have to be implemented! I can already accept incoming connections at the HCI level, but there is much more to come, such as the service discovery protocol.
Blunt architecture
Bluetooth support will consist of two main pieces: The RFCOMM communications layer (realized as a comm tool) and the Bluetooth Setup application. The RFCOMM layer will handle HCI and L2CAP to provide a serial port over Bluetooth. The Bluetooth Setup application will handle the rest, including discovery and pairing. It can actually be implemented in NewtonScript, I’ll have to see about that.
There is also the area of Bluetooth by-products. At the moment, I’m thinking mostly about location-based services. It would be quite simple to let the Newton scan its Bluetooth neighborhood upon wakeup to determine where it is at the moment. It is also easy to advertise its presence by simply being discoverable for some time. All this could be done at the NewtonScript level.
Unfortunately, the current architecture restricts usage of the serial port to one program at a time. As an example, a location-based service would have to share the port with the receiving transport and the sending transport (Neo). Unless I figure out something smart such as multiplexing at the Endpoint level…
Blunt
Blunt - the Bluetooth driver for the Newton - is taking shape. The first fragments are now on the Experiments page. Today’s problem was how to get the HCI layer called… the original TAsyncSerTool handles input only if there is an active InputSpec on the NewtonScript side. For now, the approach is to fake that we are in the receiving state. Seems to work, at least I get periodic results from a Bluetooth discovery process.
Communication Protocols 101
One of the challenges in the Bluetooth driver project and any other project involving the NewtonOS communications framework are the communication primitives which are exposed to the NewtonScript layer.
These primitives (Bind, Connect, Input, Output etc) have to be implemented by any lower layer protocol such as RCOMM or SSL. It is of course possible to bypass them and offer a separate API via a native module (I can already talk to my Bluetooth card on that level), but then, no existing application could take advantage of the new protocol.
The RFCOMM layer has to fit under the generic TCommTool interface which in turn is called using the outer communications primitives. In addition to that, there is a tradeoff in reusing the existing code for interacting with the serial chip (implementing in TSerTool and TAsyncSerTool) and being able to implement whatever buffering scheme and packet handler is necessary for the HCI/L2CAP layer.
I figure that reusing the TAsyncSerTool is the best way to go. It implements a significant amount of logic and I don’t think I could reimplement a comm tool from scratch. This means that I have to reverse engineer it to that extent that I know how to fit the HCI/L2CAP layers into the existing code.
From the experiences with Nitro I have already figured out which functions are called in which order when a NewtonScript client uses an endpoint. The situation regarding the RFCOMM tool is quite similar: The TSerTool handles the upper layer input buffers which are visible to the NewtonScript layer. The TAsyncSerTool uses its own ring buffers to talk to the hardware and fill the TSerTool buffers. This is point where the RFCOMM tool will come into play. Instead of simply copying the data between the buffers, it will have to implement the HCI/L2CAP logic. This requires changes to the IRQ handler, the DoInput and DoOutput functions and a number of internal functions (which are thankfully all declared as virtual so they can be overloaded).
The biggest difficulty is to implement the logic which the NewtonScript glue layer expects from the comm tool layer…
I thought I’d share this
Look at the Experiments page. Courier and ntox 2.0 are still not finished and tested enough, but surely, somebody will find them at least interesting ;)
The plan
It seems that many of my assumptions about a Bluetooth driver for the Newton were correct and it is indeed possible to implement such a thing. The basic idea is to implement a new comm tool based on the TAsyncSerTool which would provide the RFCOMM layer. This is very similar to the work I’ve done on Nitro, it might even be a bit cleaner … and run on on older MessagePads as well.
One common low-level interface to Bluetooth cards is a UART. This means that we can simply talk to the card via a serial channel. There are different kinds of UARTs - if a card would use a UART the built-in TSerialChip driver won’t recognize, we would have to implement a new TSerialChip driver. But that is fortunately not required, the Newton seems to recognize most UARTs just fine.
The next layer is the Bluetooth HCI layer. It is a set of commands and events encapsulated in data packets. It provides functions to establish connections, do the key and PIN code negotiations, transmit data and so forth.
On top of the HCI layer goes the L2CAP layer. It provides logical connections over a physcal HCI connection. This will be the basis for the RFCOMM layer.
“A communications card has been inserted” …
… let the fun begin! The card is a PICO PCMCIA Type II Bluetooth card. Currently, there are large numbers being sold on eBay in Germany, so I invested the donations I’ve received so far (thanks guys!) and got one for myself.
This means that I can start with the probably insane task of implementing a Bluetooth stack for the Newton. At least the RFCOMM part…
As for the cards that finally might be supported: They have to be PCMCIA Type II cards (16 bit, 5 Volt - 3.3 Volt might work too). CardBus cards such as the newer Belkin models won’t work.
Some weird stuff
I’m not sure if it is useful for anybody, but I just uploaded my old serial test code and the modified Endpoint code. It’s under Various Experiments.
The Neo/Nitro split
Since syncing turned out to be less satisfactory than I thought, I started to add the missing TCP/IP transport to Nitro. Taking the OBEX layer out of Nitro was a logical consequence, with Neo as the result. The TinyTP/IrCOMM part of Nitro has been stable for quite a while now, warranting a 1.0 version number. Taking OBEX out of Nitro also make bug tracking easier.
Neo contains the previously released IrOBEX transport and a new TCPOBEX transport. The TCP transport is far from perfect (no status dialogs, no receiving), but it allows you to send vCard, iCalendar and other data to a desktop computer. Too bad that the available OBEX solutions for desktop computers are a bit unfinished… But OpenOBEX is good enough for that task.
Later, Neo might also contain an RFOBEX transport, if and when there is Bluetooth support on the Newton. We’ll see…
IrDA, IrOBEX, Nitro, IC/VC and the Newton
Nitro and IC/VC try to follow the spirit of the original Newton architecture by reusing the existing interfaces you already know (hopefully): The Transports, the Inbox, the Action button and the Routing Slips. Here’s an overview:
- A transport is responsible to send and receive data. Transports are for example Mail, Beam or IrOBEX. You chose a transport via the Action button (the one with the Envelope)
- The Inbox is used by transports to store incoming data. You can tell which transport put something in the Inbox by looking at the item’s icon. If there is an application on the Newton that knows what to do with the item, it will add a “Put Away” choice to the menu behind the Inbox button with the tag on it.
- The Action button is used to do something to the currently selected item in an application. There are three groups of functions available: Things like Duplicate and Delete, sending via a transport and application defined actions. Sending via a transport works by selecting the item you want to send, choosing the transport from the Action button menu and specifying the sending options in the Routing Slip.
- The Routing slip is used by transports to get more information necessary for sending an item. This might include the address where to send to (for Mail) or the data format in which the item shall be sent.
Now how does all this apply to IrDA et al.? Quite simple:
- The transport which Nitro provides is called IrOBEX. IrOBEX is a standard understood by most IrDA capable devices. It is quite similar to HTTP.
- If you want to receive items via IrOBEX, you’d use the Inbox. There, IrOBEX is offered as a choice in the Receive menu. If you’re receiving things from a Windows PC, you have to modify the IrOBEX preferences (accessible in the Inbox [i ] menu) to use OBEX:IrXfer. This is because Microsoft sucks and doesn’t like to conform to standards.
- The Action button is what you use to send items via IrOBEX. Just select the item to be sent and chose IrOBEX from the Action button menu.
- The Routing slip comes into play if you have installed IC/VC. IC/VC offers the iCalendar/vCalendar and vCard data formats. You should use these if you want to import the data into the peer devices’s applications, e.g. an address book or calendar application.
- IC/VC also adds the capability to put away vCalendar, iCalendar, vCard and Newton packages from the Inbox after they have been received via IrOBEX. Please check the IC/VC preferences in the Inbox (under the [i ] menu) to specify what should happen if there are conflicts between received items and items already on the Newton.
Some examples:
- You want to send a Names entry to your T68i. You open Names and select the entry to be sent. Then you tap the Action button and select IrOBEX. Because you want to use the entry in the address book on the phone, you select vCard as the data format in the Routing slip. Then you tap on Send, select Now and point your Newton to the T68i which has IrDA reception enabled.
- You want to receive a calendar entry from your phone. First, you open the Inbox. Then you select IrOBEX from the Receive menu. Point your phone to the Newton and initiate sending from the phone. After the item has been received, tap on the Tag button and chose Put Away. In the following dialog, chose IC/VC as the target.
- You want to export all your Names entries to your desktop PC via email. You select all Names entries in Names, tap on the Action button and chose Mail as the transport. You fill out the necessary fields in the following Routing slip, select vCard as the data format and tap on Send. The email will be one vCard file which you can save as one file on the desktop PC and import it into programs that know how to deal with vCard files. Microsoft programs don’t fall into this category because they can’t handle more than one vCard in one file (there are however workarounds).
In no particular order
I think I have to write this down so I don’t forget it:
- Improve Patch for NIE Modem and Serial Support
- Generic Package installer for Inbox (part of IrOBEX?): check.
- New UI for MAD Max
- Fix MAD Max truncated playback bug (schedule shorter sounds?): check.
- SMS sending transport to be used with Nitro
- Delay automatic disconnect on no IAS activity in Nitro
- MasterCook MX2 file reader: check.
- C++-based HTTP library for improved streaming
- Reverse engineer full C++ codec interface to schedule sounds from C++
- Reverse engineer AIFtoNTK to use ARM tools
- Use ARM tools to compile MAD Newton
- Port faac (AAC decoder) and Tremor (Ogg Vorbis decoder)
- Fast HTML renderer plugin for NetHopper: check (Courier)
- Speaker dependent speech recognition using EARS
- Bluetooth CommTool driver for RFCOMM
- Time and position dependent application scripter
- FAT/FAT32 file reading in MAD Max via ATA support
- Buy lots of chocolate and/or beer
Something completely different
… well, not really. But I took a closer look at the latest ARM compilers to find out how to make them work with the Newton tools. The advantage of using the ARM compilers are probably better code generation and definitely the support for StrongARM specific opcodes. The most interesting case are the 64 bit multiplication operations and support for 64 bit integers which are more or less required for optimized MP3 and Ogg decoding (you see where I’m getting at…)
Turns out that it is not impossible to use the new compilers, but also not trivial. The first hurdle is of course how to get the compilers. Thankfully, I can play with the latest tools at work, including the ADS tools. But the next step is a bit more challenging. The ARM tools are a distinct relative of the Norcroft tools used by the NCT. They differ however in the file formats they use. The ARM tools are nowadays ELF-based, whereas the Norcroft tools used AIF and AOF as object formats. This requires that the complete compilation and linking process has to be done with the same tool chain. Switching file formats in between stages doesn’t really work that well…
The last step in that process is the conversion from an AIF image to the format used by the NTK. This step is done by the AIFtoNTK tool that unfortunately does not like the AIF image created by the new ARMLink and FromELF tools. Add the different debugging symbols to the mix and you end up with a total mess. But all is not lost, the AIFtoNTK tool might be easy to reverse engineer (it is only a couple of dozen kilobytes large). It takes the code, relocation and debugging symbol areas from the AIF and puts them in an NTK wrapper. That shouldn’t be too hard to reimplement…
However, there are still quite a few other projects I’d like to improve before I get back to this.
Using Nitro
Since I’ve got the GPRS settings for my Nokia 3650 correct, I was able to use Nitro a bit more extensively. This included using Raissa, SimpleMail, Nethopper and NewtScape. Overall, it went quite smoothly, no crashes and more or less acceptable speed (the 115kbps figure for IrDA is just theoretical, the protocol overhead is quite big).
It seems that at least for me, the combination of a Newton and a cell phone covers most use cases. It is actually strange that the situation has gotten better recently with the rise of RSS. It allows me to fetch information in a compact way, without having to visit huge, feature-laden web sites… but I still have to put together a package with some sample RSS channels.
[if you looking for a tutorial about using Nitro and IC/VC, here it is. ]
Why it works the way it works
With the release of IC/VC (the iCalendar/vCard route layer), some explanation of the used mechanisms might be helpful. The button with the little envelope which you see in your Newton applications is actually a very powerful tool. It is a system wide context menu that allows to perform actions based on the type of data currently displayed. The actions are either application defined or available throughout the system. The latter ones are clearly more interesting.
Most Newton applications follow the same principle. They display a list of items which are stored in a soup and can be modified individually. Every item is contained in a frame and typed via the ‘class slot. The system wide actions are available based on this type. The most common use for these system wide actions is to send the current item via a transport to somewhere else. Other actions are possible as well (e.g. reading an item via MacInTalk), but they are also based on the item’s type.
I’ll cover the transport case here. A transport advertises which data types it can handle. The most common data types are ’text and ‘frame, but there can also be others. Indeed, if you look at the Names or Dates soup, you’ll notice that there are ‘person, ‘company and ‘meeting types. Does the transport have to know about all of these? Not at all. It would be quite difficult to implement for example an email transport that would have to translate every possible data type on the Newton into an email message.
Instead, route scripts are used. They provide the translation between the application specific data types and the more general “transportable” data types. IC/VC is an example of such a route script. It registers as a handler for the data types used by Names and Dates and provides them in a form Nitro can handle them. The OBEX part of Nitro declares that it will handle the ’text datatype, so that’s what IC/VC provides. With a little twist: types on the Newton can have subtypes, and IC/VC does not provide an object of the type ’text, but instead ’text/calendar or ’text/directory. The neat thing about that is that it is still text and can be send via any transport that handles text. Nitro takes the subtype into account and sets the appropriate OBEX headers according to that.
So what happens now when a user taps the action button? First, the system adds all the application defined actions to the popup menu. Then, it identifies the type of the current object (I’ll spare you the details what happens in an app’s overview). Based on that type, it constructs a list of route scripts that can handle the data type. The final step is to then determine which data type each route script will provide and compile a list of transports that will handle these data types. These transports will be added to the popup menu.
The double indirection of route scripts and transports is really well done and is one of the things that puts the Newton OS still so much ahead of the rest of the PDA world. And that five years after is discontinuation…
Timing is crucial
Windows is again driving me crazy. First is the stupid default usage of OBEX:IrXfer instead of OBEX, including undocumented error codes, and now it’s the timing when sending items to the Newton. Well, the Newton IrDA stack has some flaws too, but only with Windows it’s so tricky. The issue is receiving items which works now a bit differently than the standard procedure (at least on Windows 2000). The problem is that the Newton listens for an incoming connection, accepts it, waits for some action and disconnects if nothing happens.
Turns out that the way that Windows was designed to work clashes with this. If you initiate reception on the Newton, the IrDA activity icon is shown on the Windows task bar. Only then the option to send an item will be added to the context menu in the Explorer. By the time you have waited for that and activated it, the Newton as already disconnected, thinking the peer fell asleep.
The workaround is to open the wireless file transfer panel on the Windows side before that, navigating to the file to be transferred, initiating the reception on the Newton and then sending the item from Windows… not nice, but it works - besides the fact that there is probably still a protocol error in Nitro that I have to fix.
That squishing sound…
… you are hearing are bugs being eliminated. One by one, but going forward steadily. I just uploaded Nitro 0.2.2 which should be more robust and compatible. The most important change in the Nitro core is that no more out-of-band TinyTP and IrCOMM packets are sent. This might complicate matters in those cases where the peer runs out of packets or wants to use the simulated modem lines, but it prevents Nitro from crashing peer devices due to malformed packets. I might have to look into that later and re-enable these packets, but for now, I can reliably use my 3650 as a cell phone modem.
The second bigger change was the addition of the OBEX:IrXfer option for sending and receiving data. It is needed because the Newton can only advertise one IrDA class when listening, and in some cases, there is a mismatch between the class the peer device expects and the class the Newton advertises (e.g. when sending from Windows 98). The correct behaviour on the Newton side would be to allow more classes but there seems to be a hardcoded logic that only the class that has actually been set via the NewtonScript endpoint interface will be accepted. Other classes and attributes can be added to the IAS server, but they are ignored for the LSAP selector (which determines the class).
Some bugs have been fixed regarding handling of OBEX data, hopefully increasing compatibility…
Handling Unicode
One very nice thing about the Newton OS is that it is fully Unicode-based. Unfortunately, this raises some issues when communicating with other devices which are not Unicode-aware… in addition to that, Unicode comes in different flavors which all want to be dealt with.
Handling Unicode in IrOBEX transactions requires actions when sending and receiving data that has a textual representation. This includes plain text, HTML, iCalendar and vCard data. My initial strategy for the sending part is that all text based routing data (e.g. all the textual representations generated by the standard route scripts) will be sent as Unicode, to be more specific, UTF-16 Big Endian. This is the native format on the Newton. Other routing formats such as vCard and iCalendar will most likely be UTF-8. Hopefully the receiving end of this will handle that accordingly…
Receiving data is probably a bit easier, I have already implemented a function to recognize and convert ISO8859, UTF-16LE and UTF-16BE for text data. Memory consumption is however a concern, that is not optimized yet.
Nitro 0.2.1
Just finished the latest version of Nitro. Quite some work has gone into the OBEX layer, but it is indeed a nice test of the lower layers of Nitro as well. There are still bugs in that lower layer that can cause nasty crashes or stalled communication, but I hope to get those ironed out eventually. One interesting discovery I made while debugging with Hammer is that the IAS implementation on the Newton is actually quite complete. IAS is used to advertise the IrDA services and parameters to peer devices, one example are the device name but also IrCOMM parameters. The part that made people believe the IrDA stack on the Newton is incomplete is only the NewtonScript interface. The rule to remember here: You don’t know anything about a system unless you have really gone down to the metal!
The OBEX implementation in Nitro can be used also via other transports. It might be interesting to use TCP/IP or a serial connection. At least for Unix, there is Open OBEX that could be used on the peer side.
Clearly the next step after fixing the bugs is to implement the missing route scripts. Beaming text back and forth gets boring after some time! Sending iCalendar and vCard data is probably very easy, and some initial tests have shown that the routing stuff on the Newton is very well thought out. As soon as a route script implements the vCard or iCalendar conversion, it can be used over any transport, making it e.g. possible to send iCalendar entries via email.
Tedious Transport
The whole high level communications infrastructure on the Newton is fairly complex. It takes place mostly in the NewtonScript layer and is therefore built on cooperative multitasking: Instead of using threads and semaphores, callbacks are used. This requires much more careful design…
Anyway, the OBEX transport now works over IrDA. It does not handle error cases gracefully and contains no user feedback. But it is integrated in the Newton’s I/O box mechanism so sending and receiving over IrOBEX just looks like any other transport (e.g. email). I can currently send text files to my cell phone or a desktop PC. This includes everything that has a textual representation on the Newton: Notes, Names, RSS feeds, … Receiving files is not implemented yet but is hopefully not too difficult.
It will get interesting as soon as more routing formats are supported. iCalendar and vCard are one example. Routing formats are independent from the transport and can be used over any transport. The sending part of iCalendar and vCard is next on my todo list.
BEGIN:VCALENDAR…
… that’s the output I just got in my Inspector window. Communication via IrOBEX between my Newton and a Nokia 3650 works. Implemented at the moment are an OBEX server with the Put method and an OBEX client with Get and Put methods. OBEX is transport neutral, but as I wrote before, it is a nice test case for Nitro. It actually confirmed that I can set up a listening TinyTP endpoint (works for OBEX but probably not for IrCOMM which is of no big use in that case anyway) which I haven’t tried before.
This is the plan: Wrap the OBEX client and server into a Transport for sending and receiving any file type via IrDA and TCP/IP. Implement a Routing layer for iCalendar and vCard datatypes (encoding and decoding), including a merge heuristic for incoming iCalendar and vCard data. Result: Making my Newton more usable in daily life, effectively saving its butt - I’m still resisting to buy a cheap Zaurus ;)
Controlled Crashes…
… are something nice. There was an interesting problem with Nitro when connecting to a Windows 2000 machine, causing an immediate crash. This was easy to reproduce and catch with Hammer. The problem was an IAP query frame sent to the Newton after successful connection. It seems that the lower levels of the Newton’s IrDA stack can’t handle this. The IAP frame was sent because the IrDA stack on Windows 2000 constantly sends out active connection requests which of course contain these frames. If the Newton also initiates a connection, then there is a chance that these frames get received at the wrong time in the protocol.
The solution is to defuse the stray IAP frame and turn it into a innocent IAP acknowledgement frame. I’ve still got to test this further but it doesn’t seem to cause any damage. The result is uploaded as version 0.1.2. It would be nice if people who encountered crashes upon connection could try if it helps.
Version 0.1.2 also contains code to connect to other peer classes such as OBEX or IrLPT. This is based on the connection options passed to the NewtonScript endpoint. Nitro Test demonstrates this feature.
Some Updates
I’ve been busy working on OBEX (currently over TCP/IP, but it is transport independent). OBEX should be a valid test case for the TinyTP layer of Nitro. Nitro seems to be working best on Siemens phones and worst on Ericsson models. Testing this is pretty complicated since the means to signal an error condition (or even logging some data) from within a comm tool are quite limited. For now, the focus will be on catching potentially critical states in the protocol and not do something stupid.
Interestingly enough, Nitro seems to work my Nokia 3650 at least to some extent. It connects via a regular GSM data connection, but then the phone crashes after transferring some data. I’ll have to try that with a more recent firmware for the Nokia.
I’ve also been able to finally try a USB/IrDA adapter under Mac OS X 10.1.5, and unfortunately, it doesn’t work at all. I suspect that this is due to the adapter (an Actisys IR4000US), which isn’t even supported under Linux. Other adapters might work - Apple is even selling them via their European web store.
Bugs and Crashes
Now that the IrCOMM code will be used more widely, there should be an explanation about the current bugs and why crashes occur.
The most common symptoms of something going wrong are:
- a spontaneous restart of the Newton
- an error message saying “Sorry a problem has occurred” with a positive, random error number
- stalled communication between the Newton and the peer device
- crash of the peer device
- no connection to the peer device
- failure of an upper level protocol (PPP, TCP/IP, HTTP)
Symptoms 1 and 2 hint at a mistreatment of some of the internal IrDA data. One example is the treatment of a IrLMP control packet as a data packet. I’ve tried to make the code robust for these cases but it depends mostly on the packet types sent by the peer device (and thus on the peer’s IrDA stack implementation) and the level of my successful reverse engineering of the IrDA and comm layer.
Symptom 3 happens when there is either a protocol error (i.e. one side sends packets the other side doesn’t handle in its current state) or, more likely, there are not enough resources on the Newton to deal with the packet. A typical situation is receiving lots of small packets and handling each packet in a time consuming way. The layers of the comm system are not completely decoupled, meaning that an upper layer can effectively block the whole stack. If that happens, the Newton sends RNR packets to the peer, telling that it is currently not able to handle data packets. The cure for a situation like this is to increase the number and size of the IrDA receive buffers (currently, I use seven 512 byte buffers).
Symptom 4 is causes by sending a malformed IrCOMM or TinyTP packet. Although I tried to catch all packets leaving the Newton to transform them to correct IrCOMM packets, there might still be a place somewhere where a packet is passed through an unknown channel (this happened with the PPP driver talking directly to the serial comm tool).
Symptom 5 is usually a sign that the peer device and the Newton disagree with IrDA service to use. Currently, the Newton requests the IrDA:IrCOMM service over TinyTP. Peer devices can however provide IrLPT or something else and as a consequence, deny the connection.
Symptom 6 is unrelated to IrCOMM, but it could still be caused by errors in the Nitro code, e.g. when packet data is corruptet.
Anyway, I’m working on all of these…
0.1
This is it: Nitro. It comes with source code so you can take a look at all the stuff I recently wrote about. I will also put the hacked TSerialEndpoint and the TSerialChip implementation on the web at some time.
Nitro for now advertises it’s services under the “ircm” label. Any application that wants to make an IrCOMM connection should use this as the service identifier. For applications that haven’t been modified (e.g. the NIE Modem and Serial support module), patching the application package by replacing the Unicode string “aser” or “irda” with “ircm” should do the trick (Unicode strings use two-byte characters, in this case the high byte is zero and the low byte is the ASCII code). I’ve provided a patched NIE Modem & Serial module on Nitro’s web page. I’ll try LookOut as soon as I receive my USB IrDA dongle.
Eric Schneck (of iTunes Plug-in fame) has had success using Nitro with his Motorola Timeport T280, Earthlink and SimpleMail. I have currently some trouble for establishing a PPP connection with a Nokia 3650 with Radiolinja as the provider, so this calls for some logging capabilities to find out what exactly happens.
Other missing pieces are a working OBEX layer (which shouldn’t be to difficult), the associated data handlers and a way to not only initiate but also accept incoming connections.
Success!
Task almost complete: I just got a working PPP connection via IrDA from my trusty development MP2100 to a Linux box. Including fetching a web page, so there was also some data going back and forth. There is still some serious testing left (including other devices such as cell phones), and some features like hardware flow control are missing for now, but this is definitely a nice success!
Anatomy of the TIrDATool
The latest difficulty (as if there haven’t been enough) was to find a place where to handle the IrCOMM specific data in the regular IrDA data flow. My first idea was to use the StartOutput and GetComplete funcitons as hooks. But again, the NewtonScript layer caused some trouble. To get it working anyway, the inner workings of the TIrDATool had to be exposed a bit more.
The TIrDATool class is derived from TAsyncSerTool. TAsyncSerTool provides asynchronous serial I/O via two ring buffers (TCircleBuf class), one for reading, one for writing. On the MP2x000, the ring buffers are filled by the Voyager chipset using DMA.
The two scenarios that had to be investigated further are reading and writing. In the TIrDATool class, this involves copying data from the ring buffers to the buffers used by the comm layer (and thus by the NewtonScript layer). These buffers are parameters to the StartInput and StartOutput functions. They are instances of the CBufferList class and are directly linked to the parameters given to the Input and Output functions of the associated NewtonScript endpoint. And this is also the reason why hooking into StartOutput and GetComplete doesn’t work quite well: These buffers do not reflect any of the packet-related characteristics of the IrDA data flow, they are merely holding the final streams of data.
So there has to be a way to get into the data flow from the ring buffers to the target buffers. This data transfer is done in the DoOutput and DoInput methods of TIrDATool. In fact, these methods are called for any IR data transfer, including the low level IrLMP and IrLAP packets. Quite handy.
Since there is some protocol overhead and logic for IrDA, the TIrDATool uses a number of helper classes. The most important one is TIrSIR. TIrSIR uses TIrLAPPutBuffer for any outgoing requests and contains a CBufferSegment for the incoming data. The task is now to put the IrCOMM data into the TIrLAPPutBuffer before sending and removing the IrCOMM data from the CBufferSegment after receiving.
Quick Update
Implementing the IrCOMM comm tool seems to go quite well. The first step was to get all the base classes of TIrDATool correctly declared. Lots of digging through C++ vtable declarations… I’ve got this love/have relationship with C++, at the moment I’m happy how easy it is to reverse engineer and patch it. And how cross-platform and cross-compiler it is.
Much of the added logic goes into the ConnectionComplete, StartOutput and GetComplete functions of the new comm tool. Sending the IrCOMM negotiation packet right after connecting works already, receiving the answer is however still a bit unclear. I hope that the comm tool mechanism lets me schedule something via StartInput. Otherwise, a timer event based mechanism like the one I tried with the serial endpoint event handler might be an alternative.
The NewtonOS Communications System
I guess a bit of clarification regarding the whole comms system on the Newton might be useful at this point. The big picture is still unclear at some points and might even be incorrect here and there, but this is how it is supposed to work…
A communications mechanism is encapsulated in a comm tool. There are several tools on the Newton, including the serial and IrDA tool. They are derived from the class TCommTool. For the IrDA tool, the full inheritance chain is: TUTaskWorld, TCommTool, TSerTool, TAsyncSerTool and finally TIrDATool. Comm tool classes make use of virtual functions to overload needed functionality (good for hacking too). A comm tool is running in its own task.
Comm tools are instantiated via an associated service. For the IrDA tool, this is the TIrDAService class. A service is an implementation of the TCMService protocol. When declaring a service class, the capabilities macro has to be used to state which service this class provides. This is also the parameter used when instantiating an endpoint.
Services are managed by the Communications Manager. The CM is not implemented in a single class but instead via several global functions and classes. The CMGetEndpoint function is the starting point for all endpoint communication. It triggers starting the necessary comm tool task by looking up the requested service and calling the Start method.
Endpoints and comm tools are bound together via an event handler (TEndpointEventHandler). It is important to know that an endpoint can be used by the NewtonScript task and the comm tool task, requiring synchronisation to prevent crashes.
On the lowest level, a comm tool uses hardware specific drivers to perform the actual I/O operations. The drivers can be layered, in the IrDA cases it is the TIrGlue class talking to a serial chip that controls the Ir LED. The TSerialChip protocol is an example for such a driver.
So far, I have been able to implement, replace or modify some of the more interesting classes in this picture: TSerialChip, TSerialEndpoint, TEndpointEventHandler and finally, TIrDATool. This would eventually pave the way for stuff like Bluetooth, SSL or SSH.
More Asynchronous Stuff
As it turns out, the NIE Modem & Serial support is a good test case. It uses only asynchronous function calls, including nBind and nConnect. Binding an endpoint is not an issue, but as you might remember, an IrCOMM connect requires some handshaking. Also, the connection options are a bit different.
Two problems need to be solved to make an asynchronous connect work: Find a place where to do the IrCOMM handshake and manage the IrCOMM connection options in a way that it does not break anything else. The option management is indeed quite interesting in that the class that controls a TSerialEndpoint from the NewtonScript side evaluates the result of the connect and passes it on to the NewtonScript layer. For that purpose, it uses the options objects set in the nConnect call. To make the IrCOMM stuff play nice with this mechanism, the IrCOMM connect options have to be removed before controll passes on…
The asynchronous IrCOMM handshake will be handled either before the first send or receive operation. Another possibility is staring it from the endpoint’s event handling routine.
IrCOMM working in NewtonScript
Good news: A previously dumb IrDA endpoint now knows all about TinyTP and IrCOMM. This is on the NewtonScript side of things, so if you now instantiate an IrDA endpoint, you can initiate an IrCOMM connection (notice that accepting IrCOMM connections is not done yet).
There are some things to note regarding the current implementation: If the code is active, all IrDA connections are IrCOMM connections. This will could get in the way of existing applications such as beaming. It is a bit difficult to anticipate when to use IrCOMM and when to use IrLMP (the Newton’s default). For that reason, I will add a preference setting to turn on IrCOMM manually.
It is also possible to talk to another IrCOMM peer directly using the C++ interface. This is easier because it uses synchronous communication. I’ll add a NewtonScript interface to these functions as well.
One big piece of the puzzle is however currently missing: how to make existing applications that use a plain serial connection (i.e. not IrDA) to use IrCOMM? I was hacking the NIE Modem/Serial support package a bit, changing the instantiation parameters from ‘aser’ (serial connection) to ‘irda’ (IrDA connection) and the physical location of the port from ’extr’ (external port) to ‘infr’ (IR led). It didn’t really do the job, but some parts of the IrCOMM code came already to life. Time to set some breakpoints and find out what’s going on!
Near Misses
A pretty nasty problem showed up that required going deeper into the endpoint mechanism: Asynchronous data transfer. It is needed because the NewtonScript world works based on the execution of small, non-blocking pieces of code that either return quickly or kick off some sort of background task (such as the whole endpoint infrastructure). One of the central points of data handling in a NewtonScript-based communication application is therefore the InputScript that processes incoming data received by an endpoint. There is no blocking read. This is something to get used to and confuses programmers regularily. For that reason, more “recent” example code from Apple introduced finite state machines which deal with these asynchronous events on the NewtonScript side. In the C++ world, most things are however pretty standard (one reason why a rewrite of parts of NHttpLib, Raissa and MAD Max in C++ could be very promising).
The problem in the IrCOMM case is that data reception is not handled by the nRcv function in the endpoint but in the endpoint’s event handler code. I hadn’t looked into that mechanism before, but fortunately, the event handling code is part of the TEndpoint protocol interface and can be overridden. In the process of mapping the mechanism, some of the classes and data structures had to be reverse engineered as well. It was not too painful - usually, it is possible to gain some understanding by looking at the C++ constructor and the initialization code. But nevertheless, I think that reverse engineering data structures is more tedious than trying to understand code.
Asynchronous reception of data works now. With one little problem left: sending a packet when handling an incoming packet doesn’t seem to work correctly. It is needed to advance TinyTP credits to the peer. The situation becomes critical when a large amount of data is sent from the peer. If the peer doesn’t get any credit, it just stops. The solution to this is to either find a way to send data properly in the event handler or set up some sort of idle handler in a different thread that checks the remote credit from time to time and sends a packet if needed. Time to look at more event handler code!
Connect works!
The first step in adding the necessary IrCOMM code to the modified TSerialEndpoint is done. The modified endpoint will be instantiated whenever a regular serial connection (IrDA, modem and serial) is made. In addition to that, connecting to another IrCOMM device over IrDA works. Next are send and receive and then we’re done. Connecting is probably the more critical operation anyway since it contains already sending and receiving of data.
In the first incarnation, IrCOMM will work when using a regular IrDA connection. This means that an application that wants to use IrCOMM has to request an IrDA connection. For legacy applications, there are two options: Either patch the application to use IrDA instead of serial (it could be as easy as changing four bytes in the package), or go the hard way and patch the TSerialEndpoint further so that it uses a second hidden IrDA endpoint when requesting regular serial connections…
Crazy Idea No. 1
The existing IrDA tool on OS 2.1 creates a TSerialEndpoint when an application requests an IrDA connection. As we know, there is a little more massaging necessary so that the data that is sent over this endpoint conforms to TinyTP and IrCOMM. The same goes for receiving. In addition to that, data has to be sent in packets.
After the failed attempt to make the IrCOMM implementation look like a TSerialChip, the next crazy idea is to make it look like a TSerialEndpoint. This is several layers higher in the Newton’s comm stack and contains code that is called from the NewtonScript task (making it hopefully a bit less sensitive). The pitfall here is that the TinyTP layer must be able to receive a packet while sending and vice versa. Speaking in Endpoint terms, a Snd/nSnd must be possible right after a Rcv/nRcv, and a Rcv/nRcv might be needed before a Snd/nSnd without returning to the NewtonScript task. A workaround for this might be asynchronous send and receive operations.
An endpoint on the Newton is an implementation of the TEndpoint protocol class. Creating such an implementation involves providing a class declaration and definition which will get processed by the ProtocolGen tool, resulting in assembler glue code. The packager tool then prepares the implementation so that it will be registered automatically when the containing package is loaded onto the Newton. A registered protocol implementation can be instantiated using NewByName. Protocols can also be unregistered. This provides a nice way to extend or configure the system.
The problem in the current situation is however that the existing TSerialEndpoint code is to be reused since I do not intend to reimplement the rest of its functionality. But subclassing from protocol implementation classes is not possible so we have to chose a different approach here. First, we simply define the class TSerialEndpoint as it is currently implemented in the OS. When this declaration is passed to ProtocolGen, the resulting glue code references the existing TSerialEndpoint code (of course this requires setting up a jump table similar to CardGlue.a). Second, we override the necessary send and receive functions with our code. And third, we deregister the existing TSerialEndpoint protocol and register our own.
The first and third step already work and the new code gets called. The challenge is now to weave the IrCOMM implementation into this…
Deeper and Deeper
To make things a bit more interesting, there are two different versions of the TSerialChip class (to be more precise, it is actually a protocol - I might explain the term protocol later, but it is also described in the DDK). The newer one (version 2.0) seems to override the older one completely, but it is also more complex. My original plan was to simply use the older one but that caused just some fancy crashes here and there.
Using the new verion of the TSerialChip protocol is unfortunately not so straightforward for various reasons. The most important one is that any implementation of the V2.0 protocol has to be provided with a so called capability. It tells that it does indeed support the new protocol via a static data field added by the ProtocolGen tool at compile time. The data to be set is declared in a macro in the C++ class declaration, but to figure out the actual syntax of that macro, a fun little exercise of reverse engineering the ProtocolGen tool on the Macintosh is needed. HexEdit is your friend in these situations and fortunately, the Apple engineers have used a string represenation of the grammar for ProtocolGen. So the capabilities macro goes into the class declaration without a trailing semicolon. Good to know.
The current hacked dummy TSerialChip implementation now already gets pretty far. On the way, we lost our friend the TLog class (a simple utility class by Paul Guyot that uses the NameServer for logging), probably due to some concurrency problems. But Hammer works nicely and it looks like the usual Instantate/Bind/Connect sequence works for this dummy implementation. A bit later, it crashes but that is not too scary. As opposed to the stuff I saw that was going on between the different layers of the comms system. Let’s hope that this is reentrant!
NIE and IrCOMM
Looks like we’re one step closer to getting NIE to connect via IrDA. A little testing showed that my previous assumptions on how to disguise the IrCOMM implementation as a serial chip are true. The only major showstopper might be that the Newton’s communication system does not like it to have its ears stiched to… you know what! Then, the only way out would be to port a complete IrDA stack and sign into the next lunatic asylum. But for Bluetooth, this might be a whole different story and well worth it.
Newton, meet GPRS
Funny how small and seemingly stupid things can get you excited. But my cell phone has just responded to an “AT” with a nice and clean “OK”. So I got IrCOMM working between my Nokia cell phone and the Newton. Who would have thought that after all the negative statements about the Newton’s IrDA stack.
Of course, I also had to try what happens when I send an “ATDT*99#” over to the phone. And indeed it answers with the typical PPP line garbage. Leaving me with the task of disguising the IrCOMM endpoint as a serial chip so it can be used as a modem. More testing of the flow control and buffer management is also needed. For that, playing around with IrOBEX might be nice.
Code Cleanup
Nothing special today… it is time to clean up the prototype code I made for testing TinyTP and IrCOMM. This is never nice since you go from something working to something most likely broken. Restructuring always causes this first phase of putting stupid errors in the code, fixing them and then wondering how one can be so stupid to put them in in the first place.
But it always pays off. I think that developers who are afraid of throwing out prototype code and start from scratch with a new design are more likely to run into problems later on. However, when restarting from scratch, there is one thing I really wish for: A versioning system that is smart enough to take a snapshot of my current development setup, letting me peek back to find out what I did to make things work. Maybe a small script would help that archives a whole directory tree. The difficult part in this is unfortunately to figure out what to archive. And how to archive across multiple machines…
Well, the TinyTP stuff seems to work now and I’m satisfied with the low level interface. The only open issue is to figure out the initial credit the Newton receives from the peer. It is sent in the IrLMP connection response frame, for now I just use the standard value of 14. Different stacks might have different values here, but the Symbian and Linux stacks use 14.
IrDA on 2.1 Devices (Part 3)
The most interesting parts of the IrDA stack are IrCOMM and IrOBEX. IrCOMM emulates a serial connection and IrOBEX facilitates exchange of chunks of data, similar to HTTP. IrCOMM and IrOBEX are usually implemented on top of TinyTP (there is one variant of IrCOMM that uses IrLMP directly).
The goal is to enable the Newton to use IrCOMM instead of one of the serial ports (built-in port, PCMCIA serial cards, PCMCIA modem cards) to connect to other IrCOMM enabled devices. Most interestingly, this would be cell phones which can be used as wireless modems. I don’t have any firm plans for IrOBEX, but we’ll see…
IrCOMM follows the same principle as most other communications protocols: Use an existing layer, build on top of that and put any specific control data into the data part of the underlying frame format. IrCOMM introduces a number of control structures which are mostly used for flow control. It can emulate basic serial three wire without flow control, three wire serial with XON/XOFF flow control, nine wire serial with hardware flow control, or Centronics flow control. The control parameters are put into the beginning of a TinyTP frame, just after the TinyTP credit byte. The structure is simple: first a byte that tells how big the control structure is, and then (parameter type, parameter length, parameter data)-tuples. All this is defined in the IrCOMM specification.
To make IrCOMM work on the Newton, we need to have a working TinyTP layer. As noted before, IrCOMM will most likely work only when initiating the connection. But that should be ok for our purposes. The IrCOMM layer will open a regular TinyTP session to the peer’s IrDA:IrCOMM service. The IrCOMM service we request will be nine wire serial with hardware flow control, but we’ll indicate that we won’t use flow control. Maybe at some point later it can be added, but for now, I’ll try to go without it.
The remainder of the work is to make IrCOMM available as a regular serial port. For that, we have to go under the hood of the Newton Communications Manager. Fun stuff ahead!
IrDA on 2.1 Devices (Part 2)
The most important missing parts of the Newton’s IrDA stack are TinyTP, IrCOMM and IrOBEX. Today, I’ll explain briefly TinyTP. TinyTP is the simplest protocol of the missing ones and is required for IrCOMM and IrOBEX. It ensures flow control by using so called credits that are exchanged between the peers. One single credit entitles a party to send one more IrLMP frame to the peer. If you run out of credits, you cannot send any packets any more and have to wait for a packet giving you fresh credits. The credits which are advanced to the peer are stored in the first byte of an IrLMP frame.
Implementing TinyTP on the Newton is not too difficult. It requires two things: One, the packet size has to be known to locate the credit byte (it’s always at the beginning of the packet), and two, we have to be able to send the initial credit when opening a connection. The first task is achieved by getting the packet size via the kCMOIrDAReceiveBuffers option. The second task is being done by using the kCMOIrDAConnectUserData option to pass in a single byte in the connect frame. It sets the initial credits for the peer and is typically set to 14.
A read via a TinyTP endpoint has to always request the full packet size. If there is not enough data, the endpoint will simply return a smaller packet. If we want to read more data, we have to do that in chunks the size of a packet. So some buffer management is needed. Of course also the TinyTP credit management, which comes into play when we want to send something or the peer has run out of credits while sending. But basically, as soon as we have the packet size and can send initial connection data, TinyTP can be implemented. I have put together a small test application that verifies this and it does look promising. Of course all the implementation work will be in C++ since we want to get down to the actual endpoint interface!
Next will be an explanation of IrCOMM…
IrDA on 2.1 Devices (Part 1)
There seems to be a certail level of confusion about the actual IrDA capabilities of the MP2000, MP2100 and eMate, i.e. devices running NewtonOS 2.1. To make things short: OS 2.1 implements an almost complete IrDA stack up until IrLMP.
Now the long version of the story: IrDA support can actually mean a lot of things. In practice, an IrDA stack usually consists of at least three layers: IrPHY, IrLAP and IrLMP. Since it is a stack, these are layered on top of each other, meaning that for example IrLMP sends and receives data via IrLAP but adds its own control data to it. Other interesting layers are TinyTP, IrCOMM (both used for simple data transfer) and IrOBEX (exchange of structured objects).
IrPHY is the lowest layer and handles access to the send/receive hardware. In the case of the Newton, it is most likely a serial chip of some sorts connected to the IR diode. Meaning that you send data one byte at a time. On top of IrPHY sits IrLAP, the Link Access Protocol. It introduces the notion of a frame which contains three parts: An address field to specify who is the recipient, a control field that defines the purpose of the frame and a payload area that contains the data to be transmitted.
Now to IrLMP, the Link Management Protocol. It is of special interest to us since this is what we can use on the Newton. IrLMP is probably the most complex layer in the whole stack. Among other things, it introduces the Link Multiplex Sevice (multiple logical connections over one physical connection) and the Information Access Service (what services are offered by a specific device, e.g. IrCOMM or IrOBEX). The IAS can be seen as a simple directory that can be queried to find out more about the capabilities of the peer. Is organized in key/value pairs (so called Parameters) which are associated with a specific service (aka Class).
But what do we now do with all this? Simple: The IrDA stack on the Newton can be used to connect to any other IrDA-compatible device. No exceptions. Sounds nice, doesn’t it? But there is a catch: Other devices may have difficulties initiating a connection to the Newton and requesting a service other than TinyTP and IrOBEX. The reason is that the Newton’s IrDA stack can associate only one parameter with each service it provides. And some services require more than one parameter. Also, the service that the other device may connect to must be known beforehand. But overall, this is not a total catastrophy I would say, since actively connecting from the Newton side should still work (it definitely does for TinyTP, IrCOMM and IrOBEX).
More to follow…
What is this about?
Before the real fun starts, maybe some introductory words are in order… this web log is intended to document my successful and unsuccessful attempts at hacking the Apple Newton MessagePad. The Newton is indeed a strange beast, its latest incarnation (the MP2100) so far ahead of the time when it was axed that it is still fascinating to develop for it.
But as you will see from this log, fascination often turns into frustration, mostly caused by the lack of proper development documentation. And then it is time to fire up a hex editor, a hex calculator, the Hammer debugger, dig into ancient issues of the Newton Tech Journal, scan UNNA for example code, look at Usenet posts that will soon be over a decade old and hope that at the end of the day, there is a cold beer in the fridge to stop the madness…
PS: In case you wonder, the word mottek comes from a certain German dialect which was spoken by who you could most closely describe as vagabounds. It means hammer.