The first step in adding the necessary IrCOMM code to the modified TSerialEndpoint is done. The modified endpoint will be instantiated whenever a regular serial connection (IrDA, modem and serial) is made. In addition to that, connecting to another IrCOMM device over IrDA works. Next are send and receive and then we're done. Connecting is probably the more critical operation anyway since it contains already sending and receiving of data.
In the first incarnation, IrCOMM will work when using a regular IrDA connection. This means that an application that wants to use IrCOMM has to request an IrDA connection. For legacy applications, there are two options: Either patch the application to use IrDA instead of serial (it could be as easy as changing four bytes in the package), or go the hard way and patch the TSerialEndpoint further so that it uses a second hidden IrDA endpoint when requesting regular serial connections...
The existing IrDA tool on OS 2.1 creates a TSerialEndpoint when an application requests an IrDA connection. As we know, there is a little more massaging necessary so that the data that is sent over this endpoint conforms to TinyTP and IrCOMM. The same goes for receiving. In addition to that, data has to be sent in packets.
After the failed attempt to make the IrCOMM implementation look like a TSerialChip, the next crazy idea is to make it look like a TSerialEndpoint. This is several layers higher in the Newton's comm stack and contains code that is called from the NewtonScript task (making it hopefully a bit less sensitive). The pitfall here is that the TinyTP layer must be able to receive a packet while sending and vice versa. Speaking in Endpoint terms, a Snd/nSnd must be possible right after a Rcv/nRcv, and a Rcv/nRcv might be needed before a Snd/nSnd without returning to the NewtonScript task. A workaround for this might be asynchronous send and receive operations.
An endpoint on the Newton is an implementation of the TEndpoint protocol class. Creating such an implementation involves providing a class declaration and definition which will get processed by the ProtocolGen tool, resulting in assembler glue code. The packager tool then prepares the implementation so that it will be registered automatically when the containing package is loaded onto the Newton. A registered protocol implementation can be instantiated using NewByName. Protocols can also be unregistered. This provides a nice way to extend or configure the system.
The problem in the current situation is however that the existing TSerialEndpoint code is to be reused since I do not intend to reimplement the rest of its functionality. But subclassing from protocol implementation classes is not possible so we have to chose a different approach here. First, we simply define the class TSerialEndpoint as it is currently implemented in the OS. When this declaration is passed to ProtocolGen, the resulting glue code references the existing TSerialEndpoint code (of course this requires setting up a jump table similar to CardGlue.a). Second, we override the necessary send and receive functions with our code. And third, we deregister the existing TSerialEndpoint protocol and register our own.
The first and third step already work and the new code gets called. The challenge is now to weave the IrCOMM implementation into this...
Too bad. The comm system blocks when calling serial chip code from serial chip code. Not totally unexpected, but it would have been nice if it would have worked. Consider it a nice exercise in DDK development!
But what about the IrCOMM stuff? Well, not everything is lost. To recap: IrCOMM and TinyTP already work on top of a regular TEndpoint (i.e. using it as a client). This means that after a little nice packaging of the existing code, it can be used in regular NewtonScript and C++ applications (including for example an improved OBEX transport). The drawback is the custom interface. It prevents legacy applications from seeing the IrCOMM implementation just like a regular serial port. The most interesting one is probably the NIE modem driver.
There are however still some options left to get the IrCOMM stuff into the system as a regular, well behaving citizen. More on that later.
To make things a bit more interesting, there are two different versions of the TSerialChip class (to be more precise, it is actually a protocol - I might explain the term protocol later, but it is also described in the DDK). The newer one (version 2.0) seems to override the older one completely, but it is also more complex. My original plan was to simply use the older one but that caused just some fancy crashes here and there.
Using the new verion of the TSerialChip protocol is unfortunately not so straightforward for various reasons. The most important one is that any implementation of the V2.0 protocol has to be provided with a so called capability. It tells that it does indeed support the new protocol via a static data field added by the ProtocolGen tool at compile time. The data to be set is declared in a macro in the C++ class declaration, but to figure out the actual syntax of that macro, a fun little exercise of reverse engineering the ProtocolGen tool on the Macintosh is needed. HexEdit is your friend in these situations and fortunately, the Apple engineers have used a string represenation of the grammar for ProtocolGen. So the capabilities macro goes into the class declaration without a trailing semicolon. Good to know.
The current hacked dummy TSerialChip implementation now already gets pretty far. On the way, we lost our friend the TLog class (a simple utility class by Paul Guyot that uses the NameServer for logging), probably due to some concurrency problems. But Hammer works nicely and it looks like the usual Instantate/Bind/Connect sequence works for this dummy implementation. A bit later, it crashes but that is not too scary. As opposed to the stuff I saw that was going on between the different layers of the comms system. Let's hope that this is reentrant!
Looks like we're one step closer to getting NIE to connect via IrDA. A little testing showed that my previous assumptions on how to disguise the IrCOMM implementation as a serial chip are true. The only major showstopper might be that the Newton's communication system does not like it to have its ears stiched to... you know what! Then, the only way out would be to port a complete IrDA stack and sign into the next lunatic asylum. But for Bluetooth, this might be a whole different story and well worth it.
The current IrCOMM code has to be made available to the rest of the OS by posing as a regular serial port. The communication system architecture on the Newton is layered, and the serial ports are at the very bottom. The system is managed by the CommunicationsManager (CM) that coordinates a number of communication tools. You can request an endpoint from the CM, specifying which CommTool should be used. One of these CommTools is the serial tool (others are the infrared, AppleTask etc). The serial tool knows about different serial ports via their location. Predefined locations are for example the PCMCIA slots and the external port.
A serial port driver implements the TSerialChip interface. It is not too complicated and hopefully, the IrCOMM code fits right in. But making it known to the rest of the system is a bit more difficult. First, the implementation has to be registered via the NameServer so that it can be instantiated. The Packager tool will take care of this. Then, the object has to be made known to the serial chip registry, along with the hardware location of the new port. The location can actually be anything, it will be used when requesting an Endpoint from the CM. Finally, the registered port must be added to the modem setup (the global variable ModemLocations is used for that).
All of the used classes are declared and documented in the DDK. But their exact behaviour has to be verified in many cases, as it is not clear how e.g. a TSerialChip implementation will be called from the serial tool. Hammer time.
Funny how small and seemingly stupid things can get you excited. But my cell phone has just responded to an "AT" with a nice and clean "OK". So I got IrCOMM working between my Nokia cell phone and the Newton. Who would have thought that after all the negative statements about the Newton's IrDA stack.
Of course, I also had to try what happens when I send an "ATDT*99#" over to the phone. And indeed it answers with the typical PPP line garbage. Leaving me with the task of disguising the IrCOMM endpoint as a serial chip so it can be used as a modem. More testing of the flow control and buffer management is also needed. For that, playing around with IrOBEX might be nice.
Nothing special today... it is time to clean up the prototype code I made for testing TinyTP and IrCOMM. This is never nice since you go from something working to something most likely broken. Restructuring always causes this first phase of putting stupid errors in the code, fixing them and then wondering how one can be so stupid to put them in in the first place.
But it always pays off. I think that developers who are afraid of throwing out prototype code and start from scratch with a new design are more likely to run into problems later on. However, when restarting from scratch, there is one thing I really wish for: A versioning system that is smart enough to take a snapshot of my current development setup, letting me peek back to find out what I did to make things work. Maybe a small script would help that archives a whole directory tree. The difficult part in this is unfortunately to figure out what to archive. And how to archive across multiple machines...
Well, the TinyTP stuff seems to work now and I'm satisfied with the low level interface. The only open issue is to figure out the initial credit the Newton receives from the peer. It is sent in the IrLMP connection response frame, for now I just use the standard value of 14. Different stacks might have different values here, but the Symbian and Linux stacks use 14.
The most interesting parts of the IrDA stack are IrCOMM and IrOBEX. IrCOMM emulates a serial connection and IrOBEX facilitates exchange of chunks of data, similar to HTTP. IrCOMM and IrOBEX are usually implemented on top of TinyTP (there is one variant of IrCOMM that uses IrLMP directly).
The goal is to enable the Newton to use IrCOMM instead of one of the serial ports (built-in port, PCMCIA serial cards, PCMCIA modem cards) to connect to other IrCOMM enabled devices. Most interestingly, this would be cell phones which can be used as wireless modems. I don't have any firm plans for IrOBEX, but we'll see...
IrCOMM follows the same principle as most other communications protocols: Use an existing layer, build on top of that and put any specific control data into the data part of the underlying frame format. IrCOMM introduces a number of control structures which are mostly used for flow control. It can emulate basic serial three wire without flow control, three wire serial with XON/XOFF flow control, nine wire serial with hardware flow control, or Centronics flow control. The control parameters are put into the beginning of a TinyTP frame, just after the TinyTP credit byte. The structure is simple: first a byte that tells how big the control structure is, and then (parameter type, parameter length, parameter data)-tuples. All this is defined in the IrCOMM specification.
To make IrCOMM work on the Newton, we need to have a working TinyTP layer. As noted before, IrCOMM will most likely work only when initiating the connection. But that should be ok for our purposes. The IrCOMM layer will open a regular TinyTP session to the peer's IrDA:IrCOMM service. The IrCOMM service we request will be nine wire serial with hardware flow control, but we'll indicate that we won't use flow control. Maybe at some point later it can be added, but for now, I'll try to go without it.
The remainder of the work is to make IrCOMM available as a regular serial port. For that, we have to go under the hood of the Newton Communications Manager. Fun stuff ahead!
The most important missing parts of the Newton's IrDA stack are TinyTP, IrCOMM and IrOBEX. Today, I'll explain briefly TinyTP. TinyTP is the simplest protocol of the missing ones and is required for IrCOMM and IrOBEX. It ensures flow control by using so called credits that are exchanged between the peers. One single credit entitles a party to send one more IrLMP frame to the peer. If you run out of credits, you cannot send any packets any more and have to wait for a packet giving you fresh credits. The credits which are advanced to the peer are stored in the first byte of an IrLMP frame.
Implementing TinyTP on the Newton is not too difficult. It requires two things: One, the packet size has to be known to locate the credit byte (it's always at the beginning of the packet), and two, we have to be able to send the initial credit when opening a connection. The first task is achieved by getting the packet size via the kCMOIrDAReceiveBuffers option. The second task is being done by using the kCMOIrDAConnectUserData option to pass in a single byte in the connect frame. It sets the initial credits for the peer and is typically set to 14.
A read via a TinyTP endpoint has to always request the full packet size. If there is not enough data, the endpoint will simply return a smaller packet. If we want to read more data, we have to do that in chunks the size of a packet. So some buffer management is needed. Of course also the TinyTP credit management, which comes into play when we want to send something or the peer has run out of credits while sending. But basically, as soon as we have the packet size and can send initial connection data, TinyTP can be implemented. I have put together a small test application that verifies this and it does look promising. Of course all the implementation work will be in C++ since we want to get down to the actual endpoint interface!
Next will be an explanation of IrCOMM...
There seems to be a certail level of confusion about the actual IrDA capabilities of the MP2000, MP2100 and eMate, i.e. devices running NewtonOS 2.1. To make things short: OS 2.1 implements an almost complete IrDA stack up until IrLMP.
Now the long version of the story: IrDA support can actually mean a lot of things. In practice, an IrDA stack usually consists of at least three layers: IrPHY, IrLAP and IrLMP. Since it is a stack, these are layered on top of each other, meaning that for example IrLMP sends and receives data via IrLAP but adds its own control data to it. Other interesting layers are TinyTP, IrCOMM (both used for simple data transfer) and IrOBEX (exchange of structured objects).
IrPHY is the lowest layer and handles access to the send/receive hardware. In the case of the Newton, it is most likely a serial chip of some sorts connected to the IR diode. Meaning that you send data one byte at a time. On top of IrPHY sits IrLAP, the Link Access Protocol. It introduces the notion of a frame which contains three parts: An address field to specify who is the recipient, a control field that defines the purpose of the frame and a payload area that contains the data to be transmitted.
Now to IrLMP, the Link Management Protocol. It is of special interest to us since this is what we can use on the Newton. IrLMP is probably the most complex layer in the whole stack. Among other things, it introduces the Link Multiplex Sevice (multiple logical connections over one physical connection) and the Information Access Service (what services are offered by a specific device, e.g. IrCOMM or IrOBEX). The IAS can be seen as a simple directory that can be queried to find out more about the capabilities of the peer. Is organized in key/value pairs (so called Parameters) which are associated with a specific service (aka Class).
But what do we now do with all this? Simple: The IrDA stack on the Newton can be used to connect to any other IrDA-compatible device. No exceptions. Sounds nice, doesn't it? But there is a catch: Other devices may have difficulties initiating a connection to the Newton and requesting a service other than TinyTP and IrOBEX. The reason is that the Newton's IrDA stack can associate only one parameter with each service it provides. And some services require more than one parameter. Also, the service that the other device may connect to must be known beforehand. But overall, this is not a total catastrophy I would say, since actively connecting from the Newton side should still work (it definitely does for TinyTP, IrCOMM and IrOBEX).
More to follow...
Before the real fun starts, maybe some introductory words are in order... this web log is intended to document my successful and unsuccessful attempts at hacking the Apple Newton MessagePad. The Newton is indeed a strange beast, its latest incarnation (the MP2100) so far ahead of the time when it was axed that it is still fascinating to develop for it.
But as you will see from this log, fascination often turns into frustration, mostly caused by the lack of proper development documentation. And then it is time to fire up a hex editor, a hex calculator, the Hammer debugger, dig into ancient issues of the Newton Tech Journal, scan UNNA for example code, look at Usenet posts that will soon be over a decade old and hope that at the end of the day, there is a cold beer in the fridge to stop the madness...
PS: In case you wonder, the word mottek comes from a certain German dialect which was spoken by who you could most closely describe as vagabounds. It means hammer.