Mottek Blog

This Changes Everything

Sun, 08 Feb 2026 17:23:53 +0200

I know, it’s a cliche phrase, but I think everything is indeed changed. Well, that’s maybe exaggerated, but in the field I’m active in, large language models (some call them artificial intelligence) are currently having a huge impact.

In my experience, they are heavily bending time used for the different phases of software engineering. Programming tasks which used to take hours or days (or even more) are now done in minutes. This allows to shift more time to other activities or allows to do more. But in some cases, it also requires more time to be spent on other phases, otherwise there will be trouble: Especially properly defined requirements and implementation validation are becoming extremely important.

One specific example concerns reverse engineering of the NewtonOS, something various people have been working on over the last twenty years and more. It is a slow process, requires a lot of time and dedication, and sometimes quite frustrating. But with LLMs, this changes - a lot. A year ago, I was still thinking we would need LLMs trained specifically on the available NewtonOS artifacts (DDK headers, ROM disassembly, ARM reference manuals), but things have changed over the last year.

I have been trying a more general approach to fill in the blanks, and it’s quite a ride. The output is still mediocre, but the more structure and data will be available, the better it will be. The approach works actually in all cases where the solution outline is mostly known:

Do not tell the LLM to solve the problem, but think how you would solve the problem, and instruct the LLM to do so.

For the reverse engineering task, this means to first put tools together which help, use the tools, evaluate the process, improve the tools, repeat. Here, this means to automate the extraction of methods, the derivation of vtables, the derivation of class layouts, and grunt work like resolving cross references. What is really interesting is to let the LLM reflect on the process and improve the tools along the way.

Graceful Degradation

Thu, 10 Jul 2025 15:26:55 +0300

Building systems which should run for multiple years while being maintainable is quite a challenge. In the embedded space, you would look for supply guarantees, but those are only part of the solution - what good is a processor for which no SDK exists anymore? Using the latest and greatest technology needs to be balanced against maintainability, in the end the lowest common denominator wins. I call this graceful degradation for software development: Starting out with a full blown setup of an advanced IDE, LSPs, coding agents, build systems, source code control, CI systems, HIL testing and whatnot, it needs to be possible to strip all these amenities away but still be able to maintain the system and fix bugs.

As an example, an embedded framework should still be usable by having a shell script compile the application, or in the worst case, do this manually from the command line. Next up would be a Makefile, then maybe CMake or PlatformIO, and so on. I’ve been playing around with this a bit, and while it does not really support the goal of learning new things, it definitely helps to build some more long lasting solutions.

And that's why we can't have nice things

Sun, 25 May 2025 10:06:59 +0300

Over the years, I’ve dabbled with a lot of different hardware and software combinations, to name a few: Own Forth implementations, GNU Ada of different vintages, C++ from ancient CFront 3.0 to C++23, OCaml 4, Zig, Rust, Go, Node/JavaScript/TypeScript and of course C - many of these on STM32, AVR, MSP430 and desktop systems.

One of my conclusions on this is that when implementing systems for longer term use, the churn of the underlying technologies is starting to get very frustrating. Some examples:

Setting up Ada cross compilers with runtimes is despite existing scripts not trivial.
Cross compiling Zig for AVR broke at some point due to an LLVM bug.
MSP430 targets are in general difficult regardless of the programming language
Anything more modern than C++03 is a gamble.
While the core of the JavaScript/Node ecosystem has a certain level of stability, dependencies change very rapidly.
Tools might not be available for all development platforms equally

Three programming languages stand out though:

Go is by design very conservative.
C will always have compilers for whatever target system.
Forth is whatever you want it to be, by design it starts from zero, and if the hardware doesn’t change, everything stays the same.

So maybe the way to build systems which need to be maintained for more than just a couple of years is to stick to one of the three above? The safest bet would be Forth, if a host/target combination is missing, just write it yourself :) Janus has the p3212 VM target, which is written in Rust, but moving that to C would be trivial.

Need for Speed

Fri, 11 Apr 2025 21:33:20 +0300

When looking at Forth, the boundaries between compiling and interpreting are very fluid. Even when interpreting, compilation is commonplace, which is one of the strong points of Forth, the system is built incrementally.

On a Harvard architecture like AVR this is unfortunately not so simple. I implemented a compiler first with CoreForth and Janus, using subroutine threading as the execution model. This allows easy debugging of core words, and results in quite fast code even without any inlining. On the other hand, it does not allow for incremental development since code can only be executed from flash. While it would be possible to write to flash, it is not entirely straightforward, especially when using a bootloader.

Thus the next step was to implement the indirectly threaded counterpart. The code generated by an ITC compiler consists (almost) only of memory addresses, and the inner loop (NEXT) can dynamically decide whether they resolve to RAM or flash. This incurs a speed penalty, ITC is in general slower, and the RAM/flash handling still adds some overhead on top. As a result, the ITC code is about 50% slower.

What is however interesting is to develop a program first in the interactive Forth style, and then when it is ready, just compile it using the STC model.

Because I can

Sun, 06 Apr 2025 09:47:40 +0300

Growing up in the golden age of home computers caused me to get infected with a virus for which I didn’t find a cure yet, and I’m not sure if I actually want to get cured. The virus causes me to be spend considerable amounts of time fiddling with smallish computer systems with the goal to understand every byte and component, and maybe even create something useful out of them. That last part is optional though, the journey is the reward :)

The most recent incident involves the ever giving Forth, and an underused Arduino Uno. Forth on a Harvard architecture is challenging exercise, but the AVR design has a couple of nice touches which makes this quite interesting. It also seems that the combination of CoreForth-0 and Janus is actually quite capable for bootstrap new systems. Meta-compilation with Forth is turtles all the way down, so once you have a working setup, it’s better to not think too much about it and just use it.

Of course the question is why, and right now the only answer I have is: Because I can :)

API First First

Sat, 08 Mar 2025 09:36:31 +0200

My journey into the land of web technologies continues in the space and time dimension for personal and professional reasons :) In the last five to ten years, my go-to approach for creating a web-based application was REST and API-first, start with the server, and then build a single-page application on top. This aligns nicely with a domain driven design, where the starting point is the domain data model.

In many cases, the resulting server was then mostly a thin wrapper around the data storage layer. Add the usual OpenAPI spec on top, serve JSON or XML, and it is a very nice and formal system. But it does require more client side logic to achieve an actual usable system, and this client side logic is where the trouble usually started, to be witnessed from the ongoing web framework churn.

It is however not a given that the boundary between backend and frontend needs to be designed in terms of exchanging typed objects. Back in the Web 1.0 days, we exchanged data formatted for rendering, and there was no formal link between that data and the domain model. Same as sending ANSI escape codes to a TTY, or using the X Window Protocol over TCP/IP :)

This raises the question whether “API first” should be automatically be the first choice, or whether there should be a checkpoint before going down that route. And if it is not “API first,” what is then first?

There there...

Tue, 21 Jan 2025 18:48:34 +0200

…little eMate, also you shall have a fix for the Y2010 patch! This last one was a bit more tricky, I didn’t read the original patch sources carefully enough and missed updating half of it. I also had to update my MacOS to 8.1 because it looks nicer :) Once I get some reports in that the patch works, I’ll update the related web pages.

On a very related note, I just learned that BeOS has had a Y2010 problem on its own. Maybe some NewtonOS code lived on in BeOS?

What happened?

Sat, 18 Jan 2025 08:16:48 +0200

I’ve been working with web technologies in three different phases over all these years. First was the tumultuous start in the early nineties where we were participating in teaching and research at the University of Essen. This was a great time, everything was fresh and new, we participated and organised conferences on web technologies, experimented and implemented. There was a feeling that the browser can be a very capable runtime environment, but it wasn’t yet clear how it would look like. My own take on where things should be going were Active Hyperlinked Documents(AHDs), and the keen reader will see which great technology inspired parts of that idea :)

In the second phase in the 2000’s, things had settled down, and a common setup was to have a server which ran PHP or Ruby on Rails and which served HTML with CSS, often with a sprinkle of JavaScript. With that stack, I developed a bunch of small tools, and was also using it for 40Hz.org. Things still happened on the server side, but slowly moved to the frontend.

I took a break from this until the late 2010’s, when I started to get curious again and wanted to update my skills. This current phase is again a very turbulent time, with the pendulum having swung full time to browser based frameworks with WASM on the extreme end. It is however also a bit tiring. Frameworks and tools come and go, many things not making it past beta, tools which look promising being abandoned for the next great idea, and technical and architectural decisions increasingly hard to make. Of course I have to say we wanted this back then, the ability to run full scale applications in the browser was one the ideas behind the above linked AHDs, so I can only blame myself :) And there are some good sides to be able to fully encapsulate an application on the client side and treat the backend simply as a data source - as an application developer, you don’t have to involve yourself with running and updating server side components to develop your code. But the overall execution of this model doesn’t look too convincing.

As a software architect I should have answers to this, but I guess a good architect is also one who knows when to say “it depends.” Maybe one line to draw is whether you are building something for the World Wide Web, or whether it is indeed a desktop or mobile application. Take for example GitHub, which I see as a web site, but which has degenerated into a sluggish application. Compare that to SourceHut, which deep linkable, fast and quick to navigate.

Bits and Pieces

Sun, 12 Jan 2025 17:32:46 +0200

Literally! Now that the biggest problems with the Y2010 problem on the Apple Newton are fixed (the patch for the German MessagePads is under testing, and eMates should be next), here are some other things that I played around with along the way.

First, as written before, it would be great to move patch creation to a more modern tool chain. It is already now possible to use tntk to create packages on macOS and Linux (Windows as well with some effort), but creating a patch would be great. A patch package consists of three parts, the ROM patch, the ROM Extension (REx) patch, and the patch loader, and tntk is in theory capable of producing multi-part packages, but in practice, there are some issues with combining the parts. It seems that the garbage collection in NEWT/0 which compiles the packages collects a bit more than I expected, resulting in a crash. Something to improve I gu ess!

Then, to get the patches onto actual hardware, I was planning to use RDCL, but I haven’t used it in a long time, and the Ruby world has moved on, so that there’s still quite a bit of work needed to update to Ruby 3.4. The alternative is zdcl, but Zig has a very fast update rate, and also here, an update to the latest version didn’t work out of the box. Next up was gdcl, and since I anyway wanted to use Go more, I did spend a bit more time to get it to at least cover package installation.

Finally, and more on topic for the actual Y2010 problem, I was wondering how to fix any issues that would go beyond the original issue, and which would require more deeper reverse engineering. I did dig out the earlier artifacts such as symbol tables and ROM listings, but I was curious about the more recent advances in the world of rev erse engineering. Specifically, AI seems like a very good fit: Using the old Norcroft compiler to create lot of sample binary code, and then training a dedicated model with it. I didn’t go as far yet, but I did come across Ghidra in the process. And that tool seems to be covering quite a few cases. I’m planning to play more with it and post my findings, even though right now it looks the immediate problems are under control :)

712000

Fri, 10 Jan 2025 19:28:56 +0200

The patch is out :) After some of my own testing and with the help of volunteers on the NewtonTalk mailing list, I uploaded Patch 712000 now. All in all it was not too difficult to update the patch for the next sixteen years of Newton happiness, but there are still some loose ends: I want to also update the German MessagePad and eMate patches, move the patch creation off from the MacOS classic toolchain, and improve documentation.

Ok so I lied

Wed, 08 Jan 2025 14:29:04 +0200

I could of course not resist the temptation to check out how my Mac museum is doing, and if I can create patch packages without BasiliskII. The museum seems fine, my Pismo PowerBook booted nicely, the batteries seem now finally dead, but otherwise it was working well. Same with my former workhorse, the G4 Cube, its only problem is that my current monitor doesn’t like the low resolution the stock graphics card produces.

Neither machine was however in the end used to work on an updated Y2010 patch, that honor does indeed go to BasiliskII. Resource forks are as usual a challenge to move across platforms, so resorted to moving StuffIt archives around, but all pieces for the patch are in place.

It took a bit to figure out how the patch package is put together, I’ll document that later, same with the actual patch approach, but the good news is that my dear development Newton JE7420LHAYL is now reporting patch version 711999, and comes alive just at the end of 2023, December 31st 11:48 to be precise. I’ll still do some more testing and then start a proper release.

The sad part is that this will be the last patch for the Y2010 problem. By 2040, the NewtonOS low level time functions will overflow, and that is a much harder problem to fix.

Rocking like it's 1990

Tue, 07 Jan 2025 15:16:07 +0200

In order to fix the recent date related problem with the Apple Newton, the first question is how to actually develop any fix. Since I’m suspecting that the problem is related to the Y2010 problem, the starting point would be the patch 711000, making the tooling setup a bit more complex.

Patch 711000 consists of a bunch of ARM assembly files, an MPW makefile and three Newton Toolkit packages. If it were only for the assembly files, the modern tool chain brought in with mosrun would work, but the NTK packages are slightly more problematic. In that sense, the options for working with the patch are:

MacOS Classic capable hardware: Either a PowerPC machine with MacOS X, or to go full retro, a 68k or PowerPC Mac (old enough to still run MacOS 8/9). In the latter category, I still have two G4 Cubes. In the former category, I used to have a Quadra 610, but that is long gone, and I doubt sorting out the connectivity issues would be fun.
An emulated 68k Mac, using BasiliskII: Not very user friendly, but very close to the actual hardware.
Using modern tools: Compiling the assembly files using mosrun, and packaging using tnkt. This is probably the most future proof setup.

The target will be Einstein, it allows for a very quick development cycle, and it can also bring some deeper insights into running code if needed.

In the end, I think I’ll go with BasiliskII for now since I’ve used that before with Blunt related work. But the other two options are interesting as well, especially reviving the G4 Cubes would be nice!

Time travel

Mon, 06 Jan 2025 17:04:24 +0200

It seems our dear friend, the Apple Newton MessagePad 2x00, has been bitten by another time related bug. While this looks like the Y2010 bug, this seems to be a new problem. I noticed that my MP2100 ran out of batteries the other day, but didn’t pay too much attention, however a lot of people are now reporting strange behavior of their devices, with freezes and dates being set randomly. And I just checked, also my MessagePad went back to 1990! It seems this happened after January 4th this year.

From what Matthias Melcher reported, it unfortunately looks like the bug is happening quite a few layers down from the Y2010 bug. That bug is an issue in the NewtonScript layer, but this new one (we need a clever name for it still) is related to timers in the OS layer. That will likely make it quite a bit harder to fix, but we’ll see. The good news is that we have Einstein now which makes any kind of experiments much easier and with less danger to actual hardware!

So let’s get to it, fire up Einstein and play a bit with dates and times…

Objectively Old

Sat, 28 Dec 2024 11:26:27 +0200

Over the years, I have created some smaller apps for the iPhone for personal use, but to install them properly, I also submitted them to the App Store (I have not yet bothered with setting up some sort of private distribution or side loading).

Apple has from time to time reminded me to keep the apps up to date, which does make sense. I wrote the oldest app around 2010, and updated it two or three times so far with only smaller changes. This time around though I wanted to to a bit more than just a cleanup, and did some more extensive cleanup - I also brought that specific app into the current decade with regards to the user interface.

Usually, there is a bit of a mental hurdle to overcome when digging into legacy code, along the lines of “what was I thinking?” In this case, I found the work however quite refreshing: The Objective-C code was not too bad to work with, and Objective-C is for me right there together with Forth and Prolog as programming languages I stumbled across early on in my life as a programmer which kept me curious.

The update went overall quite well (especially changing to ARC was very nice), and the App Store submission process has become very streamlined over the years. No more worries about signing certificates and anxiety whether the app will be accepted. Now here is to another couple of decades of updates :)

Verb Noun or Noun Verb?

Tue, 17 Dec 2024 18:56:02 +0200

When doing one change, other changes quickly follow! I’m not exactly how it started, but for a couple of days now I’ve been using Helix as my go-to editor. Helix is a terminal based modal editor written in Rust, and one of the strong points is the out of the box experience: It has support for LSP and tree-sitter enabled, and as long as a language server is installed, ending with LSP support is a breeze.

At first, the term “modal editor” gives the impression that Helix is merely a variation of vi/vim/neovim, but what sets it apart is the way to change text. Modifications and other actions work on the current selections (yes, multiple selections are supported), and there are a number of ways to select text: Select a word, a line, a tree-sitter object, visually by cursor motions, and much more. Once selected, different text modifications are possible, but likewise, it is also possible to modify the selection itself, for example to search within a selection to generate a new, smaller selection.

There is an interesting parallel to Forth, where you push the operands on the stack, and the perform operations on them. What’s not to like :)!

The only downside is that there is at the moment no plugin system, but I found working with the Rust-based Helix code actually quite nice, and it seems not too hard to extend it if needed.

That’s not enough with the changes though. The alternative git frontend (and possibly evolution) jj popped up, and wanted to be tried. In my case, being able to work on multiple changes at the same time in a simpler way than git branches was one of the main drivers. I’ve been using StGit for a while, which is a great a tool to work with sequential changes, but jj goes one step further and supports more complex commit structures. It really seems like a graph editor, and I’m curious what kind of graphical user interfaces will be built on top of it, even though I’m very likely to stick with the terminal.

Title Cleanup

Wed, 04 Dec 2024 20:05:49 +0200

I just cleaned up a long standing annoyance and removed the redundant titles from the Mottek blog posts. I think the RSS feed should not mark any of the posts as new, but you never know. If all posts pop up as unread, no worries, it’s just a format change :)

SourceHut Mirroring and CI

Tue, 03 Dec 2024 20:51:39 +0200

I’ve been testing the mirroring setup from SourceHut to GitHub, and it seems easy enough to set up. On the GitHub side, fine grained access tokens allow modifications to the mirrored repositories, and on the SourceHut side, the tokens are made available to the build pipelines for each repo to be mirrored.

One interesting feature of SourceHut is the integration of mailing lists, git repos, and the CI system. The SourceHut build system will automatically build patches sent to a mailing list which refer to a git repo in the same SourceHut project. To avoid misuse, the build system is only available to paying users, and patches are built with the author’s permissions, not the permission of the owner of the target repo. If a patch submitter does not have a paid plan, SourceHut will not build the patch. And even more important: Patches are built without access to the receiver’s tokens.

Testing SourceHut

Fri, 22 Nov 2024 13:11:39 +0200

The next chapter of source code management has started. After SourceForge, GitHub and GitLab, I am now trying out SourceHut. The first chapters were pretty self evident, SourceForge was at the time back then the only viable option, but once GitHub came around, it offered improvements specifically in the handling of Git repos, and the networking effects.

Two downsides with GitHub led me to try GitLab (which I also started to use in my day jobs): GitHub Actions are a bit too complex for my taste, and the flat structure makes it difficult to logically group together related repos. GitLab provides a very powerful CI, and the ability to use groups and projects brings a nice structure to one’s work.

After using GitLab for a while though it does seem to be a better fit for corporate usage than for personal projects. It offers a complete and integrated solution for many development related tasks (issue tracking, CI, security scanning, deployments, …), which makes sense in a corporate setting, but my preference is the one tool for one job approach.

SourceHut is in that respect an interesting take on source code management. It does look less integrated than the other offerings, but that is not neccessarily a weakness, since it is explicitly designed that way, it is also easier to integrate with other systems. Two areas stand out for me: The ability to link resource into projects freely, and the elegant build system which makes it very easy to interatively improve the CI setup.

The downside of having less visibility than e.g. GitHub is easily handled by mirroring repositories to GitHub.

Giving it another Go

Mon, 15 Jul 2024 09:15:48 +0300

Continuing the exploration of programming languages for implementing the Newton Docking Protocol, I’ve decided to give Go another go :) The main reasons are the ability to easily target multiple platforms, and the possibility of implementing a desktop application.

Being able to support multiple platforms is important since there are not that many Newton users out there, and the more users can benefit from a new connectivity tool, the better. Similarly, having a desktop application should make it easier for end users to keep their Newtons going.

Another aspect is that Go is maybe a bit more accessible than e.g. OCaml, and more stable than Zig, in case there are people who want to contribute or work with the project.

All of this is of course still a bit theoretical, getting started with Go again has been easy, but completing the job is another issue :)

Advent^2 of Code continued

Mon, 14 Aug 2023 10:47:54 +0300

In the earlier post about a spiced up Advent of Code challenge, I wrote about interfacing with the Apple Newton MessagePads as a nice all around task for learning and test driving programming languages.

Back in the days, Apple provided the Desktop Integration Libraries for Classic MacOS and Windows to synchronize data between desktop and MessagePad computers.

After the Newton’s cancellation, Paul Guyot, Michael Vacík and Nicolas Zinovieff set out to implement a full replacement, and adding more functionality, as well as covering also POSIX platforms in the form of the DCL.

Ruby

The DCL unfortunately does not support Windows natively, so I set out to see how something similar can be created using Ruby, resulting in the RDCL. Ruby turned out to be quite pleasant to work with, but one challenge back then was how to install a Ruby-based application, I never found the time to improve the user experience to a simple “step 1: download the app, step 2: sync with the Newton, … there is no step 3”.

I revisited the RDCL recently to check out how Ruby has progressed, and was very impressed with the Async capabilities added in Ruby 3.0.

Go

One programming language which early on got the distribution problem right is Go. Applications are single files, and can be very easily cross compiled. I implemented the most interesting parts of interfacing with a Newton in the GDCL: Encoding and decoding of NSOF data, and the Dante protocol with package installation.

Go is a very pragmatic language, not overly clever or elegant, almost boring, but it does get the job done. The module system is a bit confusing, and generics are something I would still like to explore. Overall, Go looks like a solid choice for implementing at least a command line utility to interface with the Newton.

Zig

I came across the Zig programming language looking for an improved C with cross compilation capabilities, and it fully delivered on that. Having worked for a while on more high level languages like TypeScript, Scala or OCaml, it took a bit to get back to a more low level approach (e.g. manual memory management or no asynchronous support, at least yet), but it feels just right in many ways. For me, the most compelling feature is comptime, specifically the ability to work with types at compile time, which makes generic programming really easy. In some cases, this can lead to excessive duck typing, but I definitely miss the compile time capabilities in other languages (Forth is a notable exception :) ).

I got quite far with a tool to interface with the Newton, aptly called zdcl. The development experience was very nice, and I think Zig has a bright future ahead. The 1.0 release for Zig is still a while out, and there frequent changes to the language, the standard library and the build system, thus I put zdcl on hold for now, but I’ll be happy to get back to it soon.

Another change which will definitely have an impact is the switch away from LLVM. This is not so much a problem for zdcl as macOS, Linux and Windows are tier 1 platforms, but beyond that, I found Zig also a great fit for embedded targets, and developing for them will require some changes.

Clojure

In the area of cross-platform programming languages, the JVM and Node are two very interesting foundations, as both are designed from the beginning to support multiple platforms. With Clojure, it is possible to target both (either as Clojure or ClojureScript), and I started the lower level Dante protocol implementation in both Clojure and ClojureScript. REPL based development in Clojure is a really interesting experience, where code and test are iteratively implemented, and development switches back and forth between bottom up and top down. And the focus on getting your data right and letting functions transform it fits the problem space very well.

JVM and Node based implementations still have the same issue as Ruby has, the runtime environment needs to be installed, or packaged with the application. The single executable approach of Go or Zig remains simpler for distribution and installation.

Other Candidates

I experimented with OCaml, which I really like as a general purpose functional programming language. It seems to fit the problem well, but unfortunately, targetting Windows is not yet completely straightforward.

I also started to look into C++ again, which has gone through some very fundamental changes since the original DCL was implemented. It does not direcly solve the cross platform issue, but has gotten otherwise much better in incorporating more modern programming styles. It is probably nowadays one of the more straightforward candidates for a DCL upgrade or replacement, and might be worth a second look, same as Go.

At 10:28, Janus became self aware

Tue, 13 Dec 2022 16:26:55 +0200

Janus and CoreForth-0 are now finally able to compile each other - what’s up and what’s down is a bit tricky to name in meta compilation. Janus was so far more strictly speaking a cross compiler, but with the p3216 target, Janus can now generate a binary for the p3216 virtual machine which in turn can be used to compile itself. In theory, that allows the retirement of GForth as the Forth system used to run, provided that the p3216 virtual machine and the Forth binary are available.

The changes needed for Janus were pretty minimal, and the changes to CoreForth were mostly to support whatever Janus uses to produce target images, including file handling and string handling. CoreForth got a bit closer to Forth 2012 with that, but compliance is not a target so far. Rather, one of the goals was to move to the p3216 virtual machine which can be quite easily extended - it doesn’t have an FFI, but the ‘system call’ interface is simple enough to add whatever Rust supports.

Advent^2 of Code

Sun, 01 May 2022 11:21:00 +0200

Advent of Code is a nice way to get yourself to learn new programming languages. The Newton world offers an own unique challenge in the same style, quite a bit more complex though: I’ve been using the implementation of the Newton Docking Protocol as an experimentation playground. It has a number of quite interesting challenges!

Dynamic Data Structures

Data which is sent and received is stored in the Newton Streamed Object Format. The format is pretty straightforward to encode and decode, but one aspect makes it interesting: The objects are represented as a directed graph which can have cycles. This makes memory management of decoded structures a bit more complex.

Asynchronous Code

The Newton Connection Utilities (NCU) which are the traditional way to connect a Newton to a desktop computer allow not only simple tasks such as package installation or data transfer, but allow the user to also interact with the Newton while being connected. That requires handling two tasks at the same time: Dealing with the connection protocol which allows the Newton and the connected computer to initiate messages, and dealing with user interactions.

A fully featured tool to interact with the Newton needs to therefore handle multiple tasks at the same time.

Patterns

The implementation of a communications protocol is usually based on at least two more general software design patters: The protocol stack pattern for the overall structure where different protocol layers are interacting with each other, and the finite state machine pattern which structures the behavior of a single protocol layer.

Ideally, a programming language facilitates defining reusable patterns, and even better, a pattern library exists.

Cross Platform Support

Even though the Newton is an Apple product, it was always supporting Windows alongside MacOS for connectivity, and a connectivity app should likewise support at least macOS and Windows. With macOS support, Unix support is usually not far away either.

Usage and Installation

The Newton was designed as a personal digital assistant. As such, it hides technical complexities and is easy to use. A connectivity application should at least aim to be easy to use, and having a graphical user interface would be a key enabler. Additionally, installation should be as easy as possible across all platforms.

Peer Pressure

Thu, 13 Jan 2022 20:23:34 +0200

Another great WWNC lies behind, and as before, it’s a great motivator to get started again with hacking the Newton. I wanted to pick up where I left off with porting the btstack BlueTooth stack to the Newton. Now that Matthias has merged his changes into the main Einstein branch, I was able to make some good progress to integrate the serial driver which allows to talk to a physical port on the host side.

One of the test cases for serial connectivity is to connect to the Dock application. Unfortunately I was not able to get this to work from MacOS 8 running in BasiliskII (the serial port appears to be busy), but the RDCL came again to the rescue, and I was able to download the Names soup from Einstein via a pty connection.

Working on Einstein has been a lot of fun, and we’ve made good progress adding improvements on the implementation and development process. Another fun blast from the past in this context is that I was able to add a fix for the 2010 problem also to Einstein!

The Newton Development Rush of 2020

Wed, 29 Jul 2020 16:30:02 +0300

The experiments using the original CFront 2.1 sources to buid a working C++ compiler unfortunately failed at least for now, memory management in the old days simply worked differently. Assignments to the this pointer are a bit too unconventional for a modern compiler.

On the other hand, there have been a bunch of development during this year which make that effort obsolete: First, Paul Guyot picked off where I left in 2007 (again a reference to the ancient past!) and came up with a method and tools to use a modern C++ compiler to produce binaries the Apple Newton can use. This allowed him to fill a huge gap in the functionality needed to connect to the modern internet by porting mbed TLS to the Newton.

Matthias Melcher and Paul also updated Einstein, getting it to the point where either mosrun or GCC can be used easily to compile custom ROM extensions. This allowed me to implement a simple bridge to native serial ports to test … Blunt of course :) The bridge is not nicely integrated yet into Einstein, I’m waiting to Paul and Matthias to consolidate their respective changes.

Having a native way to generate binaries still leaves the NewtonScript part uncovered, traditionally this would either require a MacOS Classic capable machine, or Basilisk II. But with NEWT/0 now finally 64 bit enabled (again thanks to Paul), I was able to update tntk to allow adding native modules to a Newton package!

There hasn’t been such a flurry of activies in the Newton world for quite a while, this feels pretty amazing (and this is not even talking about the excellent WONC conference).

13 Years Later

Sat, 29 Feb 2020 11:35:00 +0300

As I hinted a while ago, there was another yak which was shaved, very relevant to the Apple Newton: In an interesting twist of fate, my former employer Nokia got hold of Alcatel, which got hold of Lucent, which owned Bell Labs… which was the birthplace of Unix. And Unix shipped with another Bell Labs invention, C++.

So when Nokia decided to release the source of Research Unix, I dug in to see whether I could find the missing sources for CFront 2.1. And behold, this archive contains what I was missing nearly thirteen years ago!

Getting the sources to compile was another story though. It is again astonishing how little C has aged (for better or worse), and how compatible the supporting tools lex and yacc have stayed, but C++ itself has changed substantially. I initially feared that I had to bootstrap compilation with CFront 1.0, but that was not necessary, and after a couple of days, I was able to pipe C++ sources through cfront, happy to see a /* <<AT&T C++ Translator 2.1.0+ 04/01/90>> */ at the very beginning of the output :)

I have not yet started on putting everything together to actually compile a Newton C++ program into a binary, but how hard can that be ;)

The other open issue is how to distribute the compiler, even though the sources contain a statement that copyright will not be enforced for non-commercial use, it might be safest to just distibute a patch set.

Yak Shaving

Mon, 05 Aug 2019 21:07:35 +0300

The keen eye will have notice some small changes and disruptions on 40Hz and in the RSS feed. What was that about? Well, some good yak shaving, of course!

It all started with the desire to visualize data from some small sensor motes I deployed (another yak which got shaved - yes, you can buy these things, but where’s the fun in that!). Which obvioulsy needs some sort of database. And while it’s possible to just use an off the shelf database, I wanted to see how an optimized time series database could be implemented in OCaml. Turns out it’s not too difficult, but at some point, I ran into a hard to diagnose deadlock, and while debugging it, I searched for a short term replacement.

One candidate is InfluxDB, which comes with a nice Grafana data source plugin. One issue to resolve is memory consumption of the database, and while my current server would likely handle the load, I was already earlier thinking about migrating to a slightly beefier machine.

Migrating a server is never fun though, so I checked which services I would like to discard in the process. And one candidate was the PHP-based content managament system I put together a long time ago. So while the server is not migrated, I switched the content management already to Hugo. There are some rough edges, I really liked for example the ability to refer to pages in a Wiki-style, but it is also nice to simplify the server setup.

That was however not all of the Yak, stay tuned for more!

PS: Apologies for some still remaining breakage. I tried to catch broken links with redirect, and will fix issues as I’ll find them.

Surprising Combinations

Sun, 20 Jan 2019 00:00:00 +0000

After reading the report of the European Mars Lander Crash investigation (and growing frustration with C as an embedded programming language), I started to look into Ada about a year ago. The Make With Ada contest, the availability of runtimes and drivers for embedded systems, and the promise of a standardised framework for real-time programming made this a very interesting journey.

Unfortunately, my go-to MCU (STM32F051, cheap, easily available, good development boards) is very low end, and the tasking part of the AdaCore runtimes are not yet fully working on Cortex M0 MCUs: My remote sensor program would run into an error of a nested lock operation because of missing hardware spin lock support on M0 cores.

Fortunately, a solution which first seemed overly complex but which turned out to be very elegant turned up, a port of the higher level Ada runtime to FreeRTOS. FreeRTOS has very wide platform support, and runs without problem on the STM32F051.

Moving my existing code to Simon Wright’s work was surprisingly simple (proving the advantage of standardising real time concepts in the language level), only the stack sizes for the tasks had to be adjusted. After years of pointer arithmetic, buffer overflows, and casting, Ada is a refreshing change!

Warm and fuzzy feelings

Fri, 24 Aug 2018 00:00:00 +0000

You have to hand it to the Newton Community, even twenty years after the Newton was canceled, it can still generate that warm and fuzzy feeling. Hats off to Noah Leon for creating a lovely movie which captures the spirit of this group! And it is of course a great honor to be part of it, it is not very often that you get the chance to appear alongside of one of Apple’s past CEOs ;)

The thumbulator

Sun, 20 Mar 2016 00:00:00 +0000

I now pushed the thumbulator branch as the new master branch for CoreForth. The idea behind thumbulator is to have just enough of a host based virtual environment to compile Forth source, and dump the resulting binary which can then be flashed. The .elf file is therefore now just an intermediate file, it could be flashed, but contains only the core part. However, if thumbulator fails to run or produce a binary, it will silently still create a garbage binary, which won’t flash properly. I guess I need to make that part more robust :)

When writing CoreForth, I got quite far with the assembly based pseudo code approach, but at some point, I wanted to switch to high level code. Metacompilation is probably the right approach but I felt it would require a rewrite of the whole system. Same with a tethered Forth. I also was using qemu for experimenting without hardware, and the 1.0 branch of CoreForth was just converting the Forth byte code into assembly sources for later compilation via an assembler.

qemu seemed a bit heavy though, and with thumbulator I found a lightweight way to have a host based environment to produce a target binary. This works as long as there are no fancy hardware dependencies at compile time, but usually, there aren’t any ;) And if something is desperately needed, it is pretty easy to extend thumbulator.

Just checking ...

Sun, 14 Feb 2016 00:00:00 +0000

Call me a pessimist (or long term thinker), but the other day, I just had to check if my Pismo PowerBook G3 still boots, and if I can still compile the Newton Y2010 patch. The way things are going, it looks like I’ll still be using my MessagePad when the next iteration of the Y2010 patch is due (around 2026).

Si4012 and CC1101

Sat, 05 Dec 2015 00:00:00 +0000

Something completely different for a change: I recently designed a simple sensor network, and wanted to keep the cost of the sensor nodes down while using the 434MHz band for range reasons. The Silicon Labs Si4012 transmitter fit the bill very nicely, but I was already happily using a Texas Instruments CC1101 radio as the receiver.

Below are the settings for the CC1101 and Si4012, the modulation is 2-FSK and the bit rate is 38.4kpbs. First the CC1101 register settings:

    static constexpr uint8_t cc110x_default_init_values[][2] = {
            {CC1101_IOCFG0,      0x2F},
            {CC1101_IOCFG1,      0x2F},
            {CC1101_IOCFG2,      0x06},
            {CC1101_FIFOTHR,     0x47}, // RX attenuation 6dB, 33/32 byte threshold
            {CC1101_PKTLEN,      0x3D}, // 62 bytes max packet length
            {CC1101_PKTCTRL1,    0x0C}, // CRC autoflush, status append
            {CC1101_PKTCTRL0,    0x05}, // TX/RX CRC enabled, variable packet length
            {CC1101_FSCTRL1,     0x06}, // 152kHz IF frequency
            {CC1101_FREQ2,       0x10}, // 434 MHz carrier frequency
            {CC1101_FREQ1,       0xB1},
            {CC1101_FREQ0,       0x3B},
            {CC1101_MDMCFG4,     0xFA}, // 135kHz channel bandwidth
            {CC1101_MDMCFG3,     0x83}, // 38.4kbps symbol rate
            {CC1101_MDMCFG2,     0x06}, // 2-FSK, 16/16 sync word detection, carrier sense
            {CC1101_MDMCFG1,     0x42}, // 8 bytes preamble
            {CC1101_DEVIATN,     0x27}, // 11.9kHz FSK deviation
            {CC1101_MCSM1,       0x3c},
            {CC1101_MCSM0,       0x18},
            {CC1101_FOCCFG,      0x16},
            {CC1101_WORCTRL,     0xFB},
            {CC1101_FSCAL3,      0xE9},
            {CC1101_FSCAL2,      0x2A},
            {CC1101_FSCAL1,      0x00},
            {CC1101_FSCAL0,      0x1F},
            {CC1101_TEST2,       0x81},
            {CC1101_TEST1,       0x35},
            {CC1101_TEST0,       0x09},
    };

And this is the Si4012 configuration (these are properties, to be set via the SET_PROPERTY command:

    static constexpr uint8_t pa_config[7] = {0x60, 0x01, 0x71, 0x00, 0x19, 0x7d, 0xff};
    static constexpr uint8_t tx_freq[5] = {0x40, 0x19, 0xde, 0x7f, 0x60};
    static constexpr uint8_t modulation_fskdev[3] = {0x20, 0x01, 0x0b};
    static constexpr uint8_t bitrate_config[4] = {0x31, 0x01, 0x80, 0x04};
    static constexpr uint8_t chip_config[2] = {0x10, 0x00};

When sending from the Si4012, the preamble (8 x 0xaa), the sync word (0xd391) and the CRC need to be added manually to the packet.

Slightly strange combination

Tue, 03 Nov 2015 00:00:00 +0000

Continuing the praise for the J-Link SWD/JTAG adapter, it works well in even slighly odd situations. The new Intel Quark D1000 is debugged via JTAG, but uses an IO voltage of 1.8V. My old Olimex ARM-USB-TINY-H adapter doesn’t go as low, but using the J-Link with OpenOCD worked like a charm!

Things which Just Work

Wed, 29 Apr 2015 00:00:00 +0000

It’s about time to praise two tools which I’m using very frequently and which have saved my day time and again: The Saleae Logic Analyzer and the J-Link SWD/JTAG adapter .

Both tools “just work” without any fuzz, and on my development platforms Mac OS X, Linux and Windows. The Saleae comes with very easy to use software, has nicely engineered hardware, extensibility and awesome support (three years warranty!). The J-Link is maybe not as sexy, but it handles the key task of debugging, flashing and interfacing with GDB without hiccups.

For both tools, cheap clones do exist, but it is a no brainer to spend a little more money to reward the excellent work that went into them.

PS: Yes, the Apple Newton also falls into this category, but I don’t think that needs to be explicitly pointed out ;)

Little motes here and there

Sat, 26 Apr 2014 00:00:00 +0000

Since the internet of things is all the rage, I wanted to do some learning by doing, and set up an own small network to experiment a little. It seems that there are a lot of technologies to chose from, and that on all layers: Different radio technologies, different processors, different protocols and so on. I settled for now on SWAP as the protocol since it’s easy enough, but I’m using the widely available NRF24L01+ chips for the radio layer. The CPUs are a mix of MSP430’s, LPC1114’s and STM32F0’s, running either a bare metal program (MSP430 and STM32F0) or mbed (LPC1114).

It has been a very interesting experience so far, and in good hacker manner, doing things from first principle gives a lot of insights: How to get power consumption to the micro ampere level? How to build small and simple hardware designs? What are good antenna designs? What about quality of service, packet loss, etc? How to visualise data and use it in smart ways?

The biggest problem is however a very common one: What to actually build with all that stuff :)?

A Subjective Classification of MCUs

Wed, 05 Mar 2014 00:00:00 +0000

MSP430 (the smaller x2 series): Like a trusted friend or cosy small home. Limited, but things work as expected.
ATmega328: Crufty and rusty. Glad there are abstraction layers to hide that.
STM32: Shiny weapon of choice (with some rough edges here and there).

Newton Projects in Modern Times

Sun, 06 Oct 2013 00:00:00 +0000

I really like GitHub for easy hosting and version control of software project sources, but it is just not that easy to use for Newton NTK projects. Mark Mason’s story of how he open sourced Dashboard is a great read about how he mastered the challenges of resource forks, lost sources, ancient version control systems and so on to get Dashboard onto GitHub. Even more, the stuff under the obsolete files folder is pretty hilarious ;)

tntk avoids these issues by being text based, but this comes at the loss of a graphical UI builder. I’m not too bothered by this, but it appears that the easiest way to go back in time and create Newton apps the way they were intended to be implemented is to get a fast and reliable PPC based Mac, install Classic, and head back to the future ;) - and don’t forget to back up frequently!

Bluetooth related developments

Sat, 17 Aug 2013 00:00:00 +0000

Some updates from the Bluetooth side of things for the Newton: I have been able to compile Blunt 2 on Linux using mosrun, the excellent Mac OS enabled 68k emulator by Matthias Melcher. The only pieces missing are packing the compiled binary, and embedding it into a Newton package. This should finally allow to cover any scenario of Newton development without the need of Mac OS Classic.

I have also looked again into suitable Bluetooth modules, and it seems that the HC-05 module, some quite interesting options exist. The module can be purchased for 5-10$, it has an integrated antenna, and uses 3.3V as the power supply - it should be pretty easy to connect to the Newton’s internal serial connector. By default, it is configured to act as a Bluetooth slave exposing a serial port profile, which is already quite useful: In this configuration, the Dock protocol can be run over Bluetooth without changes (only the internal serial port needs to be made visible to the Dock app). It won’t allow establishing a connection from the Newton to any other device to use e.g. Neo and IC/VC, but for backup, syncing and package installation this would be good enough.

The HC-05 module should be reconfigurable though to use HCI as the Newton-side protocol, and this would make it a great candidate for Blunt. The only challenge is to implement a LPT to SPI converter and use the CSR tools to flip the configuration bit for HCI support.

Battery rebuild complete

Wed, 24 Jul 2013 00:00:00 +0000

Thanks to the awesome (both technically as well from an entertainment perspective) instructions by Frank Gruendel how to rebuild an Apple Newton batterypack, I managed to revive one of my old, flat packs. It took a while, and was at certain points quite frustrating, but definitely worth the effort. Frank is certainly right about the manufacturing tolerances, even with greatest care it seems that I managed to expand the pack in one or the other dimension to the point that the built in mechanism is not strong enough to push out the pack by itself. Thankfully, Frank has that problem also covered, and I can now enjoy 2500mAh of power :)

Power Options for the Newton

Sun, 23 Jun 2013 00:00:00 +0000

Well, options is maybe a bit overstated :) From my very first Newton on, I had been using battery packs which I can recharge with the Newton. But a couple of years back I changed my usage patterns so that I had the Newton pretty much all the time plugged in, and switched to Eneloop batteries, which I charged with a separate charger. Those batteries seem to be slowly dying now, and I’m contemplating changing back to internally chargeable batteries. This would fit my usage patterns probably better anyway, as I use the Newton nowadays mostly for journaling, idea collection and to hold reference data, not that much as an organiser which has to be switched on all the time.

Since my old battery packs are pretty much dead, I already ordered rechargeable batteries with solder tabs. Let’s see how the battery pack hacking turns out!

Color coding for code

Thu, 30 May 2013 00:00:00 +0000

One project with potential Newton use is to add a simple serial port driver over USB to the Olimexino. This is mostly to understand how such a driver works in general, and if there is a possible way to add something similar to the Newton. There are of course integrated USB to serial chips already available, and solutions for the Newton, but maybe there are some other profiles which could be implemented.

The driver is written in Forth (for quick results). During development, logging is of course a useful feature. But there’s a way not to do it, and that is to log in the interrupt handler code :). Maybe some color coding of code which is called in an interrrupt handler would be useful!

Rest in Peace, Hardy Marcia

Thu, 23 May 2013 00:00:00 +0000

With great sadness I read that Hardy Marcia , a long time Newton pioneer has passed away on May 13. His software for the Newton was very early on setting examples for functionality and usability, from which I learned a lot. It feels unfair that an obviously very great guy had to leave us early. Farewell, you will be missed.

FORTH OLIMEXINO-STM32 !

Sat, 06 Apr 2013 00:00:00 +0000

Another interesting development board is the Olimexino STM32 . It is compatible to the LeafLabs Maple, using a Wiring based library and simple IDE (very similar to the Arduino boards). I couldn’t resist and had to bootstrap CoreForth on this board as well :)

Granted, this was a bit easier since I have CoreForth already running on another close relative (the STM32-P103 , also from Olimex), so it just took a couple of hours to get the basic interpreter and compiler loop running. On the ARM SoCs it really just comes down to initializing the clocks, the GPIOs and the UART - afterwards, it’s all native Forth!

Regarding the Arduino Due, I got the interface to the embedded flash controller working, and can now enjoy a very retro block editor with the ability to preserve all work over reboots. It’s almost a full blown development system, just an ARM assembler for Forth is missing.

More toys

Mon, 25 Mar 2013 00:00:00 +0000

After changing my job last year, I have gotten much closer to hardware work, which definitely suits me. What started with CoreForth expanded into all sorts of experiments with SoCs and development boards. Initially, it was a bit difficult to settle on one development platform, but now that the Arduino Due is out, I couldn’t resist and order one, to see if I can get CoreForth running on it! The chip itself seems very capable, lots of flash and RAM, and the Arduino form factor should make it easy to experiment.

Update: Seems it was easier than I thought, after digging through the data sheet of the SAM3X8E processor on the Due, I have CoreForth up and running. I pushed the changes to a separate branch on github for now, but will merge it to the main branch once I tested this a bit more.

More Hardware Hacking

Sun, 27 Jan 2013 00:00:00 +0000

One of my early ideas for using my Cortex based development boards for more practical things was to implement a simple logic analyzer. It appeared to me that simply sampling the GPIO ports and recording changes with time stamps would be good enough for protocols like I2C or even SPI (when it’s slow enough). I did want to learn the ARM CMSIS library as well, so I forked a very nice demo project for the STM32 board on GitHub, and built a simple, SUMP compatible logic analyzer. It’s not perfect yet, e.g. there is no buffering at all, and I’m suspecting the timings are still off, but it was definitely fun to build! Next up is likely a CoreForth version ;)

Playing with Go

Sun, 18 Nov 2012 00:00:00 +0000

For a long time, I’ve been unhappy with the state of Ruby for plain standalone applications. The problem is mostly the installation of a proper Ruby environment for users of the application: For the RDCL for example Ruby 1.9 is needed, but Mac OS X ships only with version 1.8.

The main need I have for the RDCL is a simple multitasking model which allows me to implement reading and writing over a serial line to the Newton while handling other tasks. It appears that Go is able to deliver exactly that via goroutines, and I started to experiment a little with that language. It would allow delivery of ready to run binaries instead of messing with installation of an interpreter first.

Go requires a bit of a mental shift though as it is not an object oriented language in the classic sense. Usually this would not be such a big problem, but almost all of the Newton’s concept are designed and implemented in an OOP way. Let’s see how it goes, in the end this is at least a nice learning experience :)

Yes, I'm still using my Newton!

Thu, 15 Nov 2012 00:00:00 +0000

In case anybody is wondering, I am indeed still using my Newton :) Even though some work related changes made using it for GTD or other ways of tracking tasks less important, I found that the Newton is still the most suitable device for taking notes, journaling and keeping a master database of my contacts. The ability to have important thoughts and ideas written down in one place, and be able to search and tag is very neat. Cloud based implementations are not bad either, but with the Newton, I don’t have to worry about connectivity :)

Memory Low

Wed, 11 Apr 2012 00:00:00 +0000

The Cortex M3 based boards I’ve been playing around with are quite astounding, the MCUs they use as Systems-on-a-Chip have a very nice range of peripherals which really only leaves your imagination as a limit to what can be done. Being able to interface with off-the-shelf MMC cards for example, or talking to a simple LCD module is pretty neat.

The only major drawback is that those chips usually have very little RAM, even compared to the Newton ;) - eight to 64 kBytes are not exactly a lot. When using CoreForth as my experimental platform, it is therefore interesting to play around a bit with different memory management approaches. Forth itself usually likes to live in RAM since it is more powerful when self-modifying code can be used, but there are ways around this limitation.

When it comes to multitasking though (which is relatively trivial using a Cortex M3 since it pushes most registers onto the stack when an interrupt occurs), a traditional preemptive threading model is quite wasteful what comes to RAM as each thread needs a separate stack. But the revival of event based programming through Node.js led me to check local continuations , in other words, functions with multiple entry points. In this area, protothreads look particularly promising for an implementation in Forth!

Again, something completely different

Wed, 07 Mar 2012 00:00:00 +0000

The low level work on Blunt, reverse engineering the NewtonOS and digging into ARM assembly led me to a small side project in an area which has tickled my brain since a long time: Forth. It all started with a hard to grasp book by Ekkehard Floegel which I used to browse through in the local computer shop but never bought until quite recently, and a short article on the Jupiter Ace. This led to one a very simple Forth-like system on the ZX Spectrum as one of my early programming adventures, with gimmicks such as 51x32 characters and text windows. Later on, I was in need of a programming language for a automotive test system (chances are that if you were ever driving a Ford or Renault made in Cologne during the late 1990s, the door mechanism was tested by it), and reused some ideas from Forth.

My most recent adventure into Forth-land is however the one closest to the original ideas and principles. The result is CoreForth , which is a Forth-implementation running on the ARM Cortex M3. Supported platforms are qemu and the LM3S811 board, and support for STM32 based boards such as the Olimex STM32P103 is coming along nicely. It is right now really not much more than a platform to easily experiment with such boards, but having the interactivity of Forth makes investigating and using the numerous on-chip peripherals of the Cortex chips very simple!

I also added a couple of links to other projects I have started on GitHub .

Debugging and the Scientific Method

Sat, 05 Nov 2011 00:00:00 +0000

Once a program reaches a stable state in terms of architecture, design and functionality, debugging becomes the predominant activity in the development process (unless the architecture or design were flawed to begin with). The largest part of the debugging process is to find the root cause for errors, whereas the actual fixing is usually a much simpler activity: Many bugs are fixed with one line changes, or even one character changes.

Verifying Assumptions

My approach to understanding why a program fails is to identify a set of hypotheses, and then test one after another. There are a couple of important points in this process: Testing can be done via a various mechanisms, such as debugging, logging, modifying input data, modifying program code, modifying timings, or changing the execution environment. Each test should produce unambiguous results, and sequential verification is important in order to allow interpretation of those results. The hypotheses need to be simple enough and non-overlapping to reduce the set of possible root causes reliably, trying to verify too much at the same time can be quite difficult.

Reducing the Debugging Work

The actual verification process (implementing and running the tests) is usually simple but time consuming, which means that I try to automate as much as possible. But the true way to reduce debugging work is to identify the right set of hypotheses, and this is unfortunately also the toughest part of debugging. It is usually a mix of domain knowledge, intuition and a systematic approach which pay off. Domain knowledge can unfortunately only be gained by actually working with the program code, it is very rare that a problem is generic enough that it can be solved without deeper understanding about the code. Intuition however is another key component altogether: It allows to make assumptions without the deepest possible knowledge, rather, deliberate ignorance helps to prevent digging in too deep (after all, understanding a system down to the metal takes a long time), and produce hypotheses e.g. based on previous experience faster. There is of course the risk that such intuition based hypotheses can lead nowhere, and I have had many many situations where I declared victory too early only. Humility is therefore a very important parter to intuition!

Why Debuggers are Bad

Formulating a hypothesis why a program is broken requires thinking and knowledge. Using a debugger usually increases the knowledge about the inner workings of a program, but it can easily stand in the way of the thinking part when coming up with ideas about error root causes. Debuggers are good to verify mini-hypotheses when stepping through code, but they are very cumbersome at getting to the big picture and testing more complex problems (plus they are worthless for testing timing issues). My impression is that they are a popular tool because they tend to give immediate results on these mini-hypotheses, which gives the illusion of making progress. For me, far more important is to read code, work with a peer, and use tracing and logging to understand the problem.

Deliberate Breakage

One of my favourite approaches to produce hypotheses is to take the program in the two states of working and broken, and reduce the difference in implementation between them, until the smallest difference is identified as the root cause. In cases where a bug is the result of changes due to ongoing implementation work, this is very simple, git for example allows to bisect a series of changes and mark working ones and broken ones until the offending change is found.

This approach can be extended by looking at different definitions for working and broken. As an example, working does not necessarily mean that it has to apply to the same program which is broken, but it can also mean a completely different program which however shares some design or implementation with the broken program. In relation to Blunt, I have been able to test some assumptions about the packet flow by checking Blunt 1 versus Blunt 2, even though both are internally very different.

It is important to keep a very open mind when looking for these instances of a working system, having them is the only way to provide a solid foundation for further debugging, and to keep sanity. If everything is broken, it is a pretty rocky road to make any progress. It is not impossible though, and in those cases, I usually start removing parts of the broken program until at least something works, and then work backwards by adding code.

Now, how about Blunt?

All of the above also applies to reverse engineering, which is still the biggest chunk of work to get Blunt running. Reverse engineering is in some sense simpler than debugging since the code is known to work, and understanding it is achieved by disassembly, or black box testing techniques to verify assumptions about how the code works. I noticed that persistence and thoroughness really pays off in this area, and it seems that Blunt is very near to finally work as designed :) Stay tuned for more!

Dear TCommTool

Sun, 23 Oct 2011 00:00:00 +0000

What is it you really want? - I’ve been experimenting with Blunt 2, setting up an internet connection using NIE and PPP over Bluetooth, and overall it appears to be quite stable, with the only very annoying problem that the data is not handed back from Blunt to the NewtonScript layer (and thus NIE). The problem appears to be the inner working of the TCommTool class, which handles this interaction. What is not entirely clear right now is how to indicate that data has arrived for further processing. This results in apparent packet loss on the NIE level, and unnecessary packet retransmission or aborted connections. It might be the most effective to debug the data flow, but the drawback of using Hammer is that it needs USB to serial drivers installed under Mac OS Classic. Well, I’ve gotten very far without Hammer, maybe now it’s again Hammer time!

More reverse engineering results

Wed, 28 Sep 2011 00:00:00 +0000

I’ve been working with the NewtonOS sampler program a bit more, and started to work a bit more on the inter-task communication. The more detailed results are on the NewtonOS Internals pages, so far one important finding is that NewtonOS is very lightweight what comes to memory management, and relies on a flat memory model with very little access restrictions. This means that memory must be allocated and deallocated with slightly different strategies than in a more conventional OS, specifically, data is shared more openly between tasks and must be kept valid as long as any task might be using it - the kernel does not create own copies of data.

Technical Information on the NewtonOS

Wed, 21 Sep 2011 00:00:00 +0000

My “NewtonOS Sampler” is now up on github , so far it’s just a skeleton though. I’ll be using the Newton DDK to implement examples of the OS services, additionally, Walter Smith and Paul Guyot have a collection of very good documents on the OS itself.

Getting to the bottom of it

Sun, 18 Sep 2011 00:00:00 +0000

The latest changes to Blunt 2 fixed the crash when sending a large number of data blocks, which was caused by memory management issues around the TUPort class used for communication between the tasks in Blunt. More specifically, the allocation and deallocation of memory for the messages was causing problems. I still need to find the right approach though, and it is probably best done via a small test program which uses tasks, shared memory, messages and ports.

Stabilizing Blunt 2

Sat, 10 Sep 2011 00:00:00 +0000

I’ve gotten Blunt 2 quite a bit more stable, and have also added missing functionality to send larger data packets in smaller chunks. I’m testing at the moment with a Conceptronic CBT100C and a PICO card, which both seem to be working quite well with 230kbps as the serial speed.

Github continues to be very useful to manage the work and track issues (I should probably indicate in the git commits also the ID of any issue I’m fixing), and for those interested, the git repository actually contains the current working stack as an installable package in the “Bluetooth Setup.zip” file ;)

One remaining stability issue is long term stability though, which I think is related to either improper memory management for the messages sent between the server and the other parts of the system, or problems with the interrupt handlers being paged out.

Blunt 2 status

Sun, 21 Aug 2011 00:00:00 +0000

Here’s a quick update on Blunt 2. I’ve moved the code to Github and started to track issues there as well. My development environment is TextMate, and I use MPW just for compiling. The simple serial line based logging works reasonably well, and I’ve not really missed using Hammer as a debugger (Hammer is still essential for reverse engineering by executing code, for simple code analysis, I use my online database of the NewtonOS ROM).

Most problems in Blunt 2 are not in the actual Bluetooth stack, but around interfacing with the Bluetooth hardware: PC Cards are using a simple 16450 UART, and sending and receiving data is not trivial due to the lack of documentation for the serial chip interface on the Newton. I’ve switched to simple non-interrupt based sending of data, which means some performance penalty but should get around problems I’ve seen with sending bulk data. The other problematic interface is between the Bluetooth stack and the CommTool, which in turn manages the interface to the NewtonScript world. CommTools are even less documented unfortunately, so I am still lacking asynchronous sending functionality and means to accept connections instead of initiating them.

Progress is expectedly slow, on the other hand, it is nice to work with the probably best notebook Apple ever made, my trusty old Pismo :)

Picking up the Gauntlet

Thu, 04 Aug 2011 00:00:00 +0000

After lengthy GTD experiments (more on that later), programming a bit for the iPhone and 202011-01-16%bringing development tools for the Newton into the 21st century, I’m thinking it’s time to revisit [Blunt 2](/Pages/Blunt 2). I have a simplified development setup where I need to use Mac OS 9 only for compiling, and can use e.g. TextMate for editing, a simple serial line for debug output and RDCL for package installation.

I uploaded the code as a first step to github, which should allow much better tracking of changes and experiments.

Newton ROM Cross Referencer

Sun, 06 Mar 2011 00:00:00 +0000

I moved my tool to cross reference the Newton ROM now to 40Hz.org. It’s usage is quite simple, just enter the function name you want to view into the text field (use % as a wild card character) and press “View” - the resulting listing is hyper linked to allow further digging into the ROM.

Experimenting with iPhone Development

Wed, 02 Mar 2011 00:00:00 +0000

As a programmer, I’m trying to continuously learn new technologies, and playing around with iPhone development was inevitable. One aspect which motivated me goes actually back to the NeXT era, and that is Objective-C. Back in the days, we just got started with a friend and his small company to dive into C++ since it was the logical next step from C, but Objective-C and Brad Cox’ idea of Software-ICs seemed already then very appealing. It was unfortunately limited on the commercial side to NeXTStep and later OpenStep, whereas C++ was much easier to deploy.

A while ago I started therefore to experiment with iPhone app development to see what the fuzz is about. Using Objective-C is very refreshing, it is great to have a compiled but also very dynamic language. It won’t beat NewtonScript though ;). The result of my experiments is now available in the iTunes app store: a simple, free expense tracking software for personal use called Geld, including DropBox support. Enjoy!

Frankennewton

Wed, 09 Feb 2011 00:00:00 +0000

Thanks to Frank Gruendel, I’m now the proud owner of three badly treated Newton MessagePads! My intention in acquiring them was to investigate case modding options. While I do like Jonathan Ive’s original design of the MP2x00 series, there is some 90’s bulkiness which would be great to shed. The most radical option is to take just the motherboard and display and construct a new housing. But even shaving off bits and pieces from the regular case might help.

Developing Newton apps with Einstein, TextMate and tntk

Sun, 16 Jan 2011 00:00:00 +0000

In this post I want to describe my setup for developing Newton apps on Mac OS X using Einstein , TextMate and tntk.

Einstein is working very well as a replacement for an actual Newton during development. An important enabler is the AppleScript interface, which allows “remote controlling” the emulator. TextMate is my editor of choice for other development projects already, and extending it to support NewtonScript and interfacing with Einstein is very easy. And finally, tntk is able to generate Newton packages of reasonable complexity if certain limitations are kept in mind.

Setup

Setting up Einstein is straightforward, and instructions on getting a ROM file are just an internet search away. I’m using the the same screen resolution on the emulator as on an MP2100, and I install a list of very useful packages:

ViewFrame with all extensions
BugTrap
StandAlone’s PrefsCleaner
PowerPrefs

Unfortunately, BugTrap is not able to save trapped errors to the Notepad, but it is possible to use ViewFrame’s intercept functionality to break on exceptions, which is a very good replacement.

tntk does not need much setup, it just has to be somewhere in the path on Mac OS X, e.g. copied to /usr/local/bin or /usr/bin. The usage of standard NewtonOS elements like protos and constants requires a platform file, which is part of the Newton Toolkit (found in the Platforms folder after installing the NTK on the Mac and named e.g. Newton 2.1).

I’ve created a NewtonScript bundle for TextMate (a current snapshot can be downloaded from the git repository on SourceForge, the unpacked folder has to be renamed to NewtonScript.tmbundle) to enable syntax highlighting and simplify package compilation and installation. Package installation is not very elegant yet, it requires a small AppleScript named Install.scpt in the same folder where the source files under development are located. I am using right now a package specific script, but it could possibly also be simplified (Package name, symbol and path need to obviously replaced with real values):

tell application "Einstein"
	do newton script "GetRoot().%<package symbol>%:Close()"
	do newton script "SafeRemovePackage(GetPkgRef(\"<package name>\", GetStores()[0
]))"
	install package "<path to package file>"
end tell

It closes the application running in Einstein, removes the package, and then installs the new package. The TextMate bundle is configured to run the script via the shortcut Command-Shift-B.

Usage

The development cycle using tntk and Einstein is slightly different from using the NTK. One major difference is the absence of the Inspector, but ViewFrame is a very good replacement, and in some areas even superior. One big advantage is the ability to use standard source code version control systems since tntk uses text files. It has always bothered me to not have good change control when developing Newton applications.

Converting existing Newton applications is not straightforward, but possible if the original NTK project is available. The NTK allows saving the project as a text file, which can be modified into a tntk compatible source file. In general, the original developer documentation by Apple is an excellent resource to get started, and ViewFrame is very useful in understanding the inner workings of applications.

Happy hacking!

Autoparts with tntk

Sun, 09 Jan 2011 00:00:00 +0000

Autoparts for the Newton are packages which do not have any visible user interface, but add functionality behind the scenes. They are usually listed in the Extensions folder of the Extras drawer.

It is possible to use tntk to create autoparts, and this blog entry demonstrates how to add a module (which admittedly doesn’t do much) to the NiftyDrop backdrop application.

The example source can be found on SourceForge.

Project File

The tntk project file is very similar to a project file for a regular application, the only difference is that it uses the part type auto instead of form.

Below you can see that the project contains only one part, which has the type auto and is made up of only one source file, main.newt.

{
    parts: [
        {
            main: "main.newt",
            files: [
],
            type: 'auto
        }
    
],

The package name definition and the reference to the platform file are next:

    name: "NewtAutopartExample:40hz",
    platform: "/Applications/Newton/NTK 1.6.4/Platforms/Newton 2.1"
}

The platform file is part of the Newton Toolkit, and tntk uses it to find predefined constants such as proto names.

Main View

The only source file the example uses is main.cpp.

The first part of the file are common definitions of constants (the naming convention I chose is to start constants with a k, methods with an M and fields with an f):

constant kAppSymbol := '%NewtAutopartExmple:40Hz%;

constant kNiftySym := '%NiftyDrop:HyprMynd%;
constant kNiftyRegistry := '%Registry:NiftyDrop:HyprMynd%;
constant kModuleSym := '%Nifty:NewtAutopartExmple:40Hz%;

Next follows a function used to add a module to NiftyDrop:

constant kAddToRegistry := func (module) begin
    local reg := GetGlobals().(kNiftyRegistry);
    local sym := EnsureInternal (module.symbol);
    if not reg then begin
        reg := TotalClone ({
            modules: {},
            prefs: {},
            infoItems: {}
        });
        GetGlobals().(kNiftyRegistry) := reg;
    end;
    reg.modules.(sym) := module;
end;

The function creates NiftyDrop’s module registry if it doesn’t exist already, and uses the new module’s symbol as a slot name to refer to the module code.

A function to remove a module from NiftyDrop is next, it simply removes the slot from the module registry and also from the set of active modules:

constant kRemoveFromRegistry := func (sym) begin
    local reg := GetGlobals().(kNiftyRegistry);

    RemoveSlot (reg.modules, sym);
    p := GetAppPrefs (kNiftySym, {});
    SetRemove (p.activeModules, sym);
    RemoveSlot (p, sym);
    EntryChangeXmit (p, nil);
end;

Next up is the actual module view. As the other elements defined so far, it is a constant, declared for later use. The view itself is just a gray rectangle with the default size of 100x60:

constant kModuleView := {
    viewClass: clView,
    viewFlags: vVisible,
    viewFormat: vfFillGray,
    viewJustify: vjParentLeftH + vjParentTopV,
    viewBounds: {left: 0, top: 0, right: 100, bottom: 60},

NiftyDrop expects a number of additional slots, most importantly the name and symbol:

    resizeable: true,
    symbol: kModuleSym,
    name: "Example Module",
    minWidth: 100,
    minHeight: 60,
    widthIncr: 1,
    heightIncr: 1,

};

The view defined in kModuleView offers no user interaction and has no other visible elements than just a gray rectangle. It would be very simple to add new views in an array in the stepChildren slot of the view.

All of the code comes together in the install and remove scripts. The Newton Toolkit uses a slightly different naming convention and shifts some code around. Instead of the InstallScript and RemoveScript function used in the NTK, I’m using a slightly different setup below.

First, the script run at installation time adds the module to the module registry, and let’s NiftyDrop know about it:

{
    devInstallScript: func (partFrame, removeFrame) begin
        call kAddToRegistry with (kModuleView);
        if Visible (GetRoot().(kNiftySym)) then GetRoot().(kNiftySym):prefsChanged ('moduleAdded, nil);
    end,

The script run at removal time (e.g. when freezing or deleting the extension, or removing the card it is stored on) closes NiftyDrop and then removes the module from the registry:

    devRemoveScript: func(partFrame) begin
        if Visible (GetRoot().(kNiftySym)) then GetRoot().(kNiftySym):Close ();
        call kRemoveFromRegistry with (kModuleSym);
    end,

The actual installation script runs the devInstallScript and returns the removal script to the system:

    InstallScript: func (partFrame) begin
        local removeFrame := EnsureInternal ({removeScript: partFrame.devRemoveScript});
        partFrame:devInstallScript (partFrame, removeFrame);
        return removeFrame;
    end,
};

Notes

tntk is creating NewtonScript packages in a quite unique fashion using NEWT/0. Technically, a tntk program is exectuted as NewtonScript, and the return value of the program is input to the package creation step. That means that a program is not just compiled, but it generates itself. The return value is the code between the last pair of opening and closing braces in the example above, i.e. a frame containing the installation and removal scripts. These scripts in turn refer to the rest of the code.

One caveat with this process is that NEWT/0 captures the complete execution environment and returns it. Since it is executing a function, and since NewtonScript functions are proper closures, care must be taken to not pull in objects which are not needed. If in the example above the elements are not declared as constants, they will be passed to any function within the code as the environment for the function’s execution, which in extreme cases causes the NewtonScript interpreter to run out of memory.

An improvement of the package generation and compilation step would be to use NEWT/0 only for function bodies, and use a more traditional code generation process to generate the package code.

Fun with NEWT/0

Tue, 23 Nov 2010 00:00:00 +0000

One of my long standing pet peeves is the problem of not being able to use Mac OS X anymore for Newton development since the Leopard days. Basilisk II is of course an option, but it’s quite cumbersome. For that reason, I’ve been trying to put together simple compiler based on NEWT/0 and the DCL to allow at least some sort of text based development.

So far my experiments are actually quite successful, and it seems that developing Newton applications with just a text editor is not that impractical. It is in fact easier when it comes down to version control management. Some things are still missing for developing larger apps, like the ability to split the code into multiple source files, and a way to embed resources into the final package, but for simple applications (and even auto parts), we might have a way forward.

This is nowhere as ambitious as Matthias Melcher’s DyneTK project, but maybe good enough to get people interested in a bit of Newton hacking. Unfortunately I haven’t gotten around to packaging the code nicely yet, so far it’s part of the SVN repository at SourceForge under tntk. I also uploaded the Mac OS X binary itself, but there are guarantees that it would actually to something useful. It has a couple of non-functional parts, but compilation should be as simple as

tntk -c ProjectFile.nprj

Below is how a simple NewtApp would look like, first the project file which describes basic app parameters, and also points to the NTK platform file which contains common definitions needed for development (this would be saved as e.g. ProjectFile.nprj):

{
    parts: [
        {
            main: "Main.newt",
            files: [
],
        }
    
],
    name: "MiniNewtApp:40hz",
    platform: "/Applications/Newton/NTK 1.6.4/Platforms/Newton 2.1"
}

And this is the main file, according to the project file above it would need to be saved as Main.newt:

kAppSymbol := '%MiniNewtApp:40Hz%;
kAppTitle := "MiniNewtApp";
kVersion := "1.0";

defaultLayout := {
    _proto: newtLayout,
    masterSoupSlot: 'MiniNewtAppSoup,
    name: "Default",

    viewJustify: vjParentFullH + vjParentFullV,
    viewBounds: {left: 0, top: 20, right: 0, bottom: -25},

    stepChildren: [
        {
            _proto: newtEntryView,
            
            viewJustify: vjParentFullH + vjParentFullV,
            viewBounds: {left: 0, top: 0, right: 0, bottom: 0},

            stepChildren: [
                {
                    _proto: newtLabelInputLine,
                    path: 'title,
                    label: "Title",
                    labelFont: ROM_fontSystem10Bold,
                    viewJustify: vjParentFullH + vjParentTopV,
                    viewBounds: {left: 0, top: 4, right: -8, bottom: 32},
                },
            
]
        }
    
]
};

overviewLayout := {
    _proto: newtOverLayout,
    masterSoupSlot: 'MiniNewtAppSoup,
    name: "Overview",

    viewJustify: vjParentFullH + vjParentFullV,
    viewBounds: {left: 0, top: 20, right: 0, bottom: -25},

    Abstract: func(target, bbox) begin
        return MakeText(target.title, bbox.left + 2, bbox.top, bbox.right, bbox.bottom - 18);
    end
};

mainView := {
    _proto: newtApplication,
    title: kAppTitle,
    appSymbol: kAppSymbol,
    appObject: ["Task", "Tasks"
],
    appAll: "All Tasks",
    allLayouts: {
        default: defaultLayout,
        overview: overviewLayout
    },
    aboutInfo: {
        tagLine: "MiniNewtApp - a NewtApp demo",
        copyright: "(c) Eckhart Koeppen, all rights reserved",
        version: kVersion
    },
    allSoups: {
        miniNewtAppSoup: {
            _proto: newtSoup,
            soupName: "MiniNewtApp",
            soupIndices: [
],
            soupQuery: {},

            CreateBlankEntry: func() begin
                return Clone({title: "Title"});
            end
        }
    },
    stepChildren: [
        {
            _proto: newtClockFolderTab
        },
        {
            _proto: newtStatusBar,
            menuLeftButtons: [newtInfoButton
],
            menuRightButtons: [newtActionButton, newtFilingButton, newtCheckAllButton
]
        },
    
],

    viewJustify: vjParentFullH + vjParentFullV,
    viewBounds: {left: 16, top: 16, right: -16, bottom: -16},
};

{
    app: kAppSymbol,
    text: kAppTitle,
    theForm: mainView,

    devInstallScript: func (partFrame) begin
        partFrame.removeFrame := partFrame.theForm:NewtInstallScript(partFrame.theForm);
    end,
    RemoveScript: func(partFrame) begin
        partFrame.removeFrame:NewtRemoveScript(removeFrame);
    end,

    InstallScript: func (partFrame) begin
        partFrame:?devInstallScript(partFrame);
        if HasSlot(partFrame, 'devInstallScript) then RemoveSlot(partFrame, 'devInstallScript);
        partFrame.InstallScript := nil;
    end,
}

Web site moved

Mon, 18 Oct 2010 00:00:00 +0000

I noticed that this web site started to show more and more slowdowns, and I asked my current hosting provider Hurricane Electric what they could do about it. They offered to move me to another server, which initially seemed like a great idea, the new hardware was indeed faster, but then I noticed that it’s running Linux on 64 bits, which causes memory consumption to go up. That by itself is not an issue, unfortunately the site was using Apache and the Python-based MoinMoin Wiki as the CMS, and Python modules which are dynamically loaded are wasting quite a bit of memory. In the end, there was no other option to rewrite the whole setup, and PHP was the safe choice. With Kohana, I found a framework which has the right level of abstraction, and after about two weeks of hacking, things should be back to normal now :) The RSS feed might need some modifications still, but I hope I can now go back to the regularly scheduled Newton programming!

The 2008 Bug

Wed, 10 Feb 2010 00:00:00 +0000

The NewtonOS still has one problem related to the year 2010 problem : Very early in the boot sequence, the OS adjusts the real time clock to a “reasonable” value. Unfortunately, dates past 2009 are not considered reasonable, and the OS resets the date therefore to 1996.

The 2010 patch kicks in after that, and it can then only get the date to something slightly better, which is 2008. A proper fix would require to change the function which adjusts the clock, but it is not in the patchable area. It also occurs so early in the boot sequence that there is no easy way to intercept it, backup the proper clock value, and restore it later.

There are some workarounds possible, for example storing the correct time every minute when the Newton is turned on, but they require a good place for the time value. Flash memory is not ideal, a better place would be somewhere in RAM which survives resets and reboots. I’m now on the hunt for such a buffered location…

Enjoy your retirement, JE7420LHAYL

Wed, 07 Oct 2009 00:00:00 +0000

After twelve years of service, my first MP2100 is headed for retirement. It has been working very reliably over the years, but recently, one strange problem started to occur: It stopped playing sounds. I checked the usual suspects (wiring, software patches, …), however the very first Newton on this planet to play MP3 files is now mute.

This device is one of the first German MP2100s, and I bought it back in 1997 from a Swiss web store. I used to baby the Newton way too much, before I realized that it can only realize its full potential if I have it close by all the time. The case looks pretty battered now, even though all clips and doors are still in place. The backlight it maybe at 50% brightness, and the screen is slightly worn. This particular Newton has seen quite a bit of places from California to Australia, and went up all the way to 70° 54’ 48" North.

Now it is time to retire this lovely device, and start using its successor, probably one of the last produced MP2100 I bought already quite a while ago. I’m not sure if it will also see twelve years of service, but you never know :)

The case of the never ending project

Wed, 30 Sep 2009 00:00:00 +0000

One observation from using Nitch now for almost two years is that GTD is not only about projects and actions, but a higher level concept on top of projects is still needed. I added roles and goals earlier to Nitch, and while that has been helpful to bring projects and actions into focus as needed, I still ended up too opften with projects that did not have a clear outcome.

The concept of a goal looks promising in this context, and I am adding more functionality to Nitch to have goals which in turn are made of projects. That allows to use goals for more longer term planning, and restrict projects to efforts which are not longer than a month.

SourceForge improvements

Fri, 14 Aug 2009 00:00:00 +0000

SourceForge seems to be on a roll… even though things were a bit rocky due to their site redesign, I’m happy to see that the file release system now again allows direct links to packages for download, no more searching through dozens of packages! It also seems that it will soon be possible to use multiple Mercurial repositories per project, so that I can finally use a real version control system for the Newton packages.

Year 2010 problem no more ...

Sun, 26 Jul 2009 00:00:00 +0000

After more testing by brave volunteers , I’m happy to announce that all NewtonOS 2.1 devices should be ready for year 2010 and beyond. In the patch collection, we now have:

Patch 71J059 : Patch for the US MessagePad 2000/2100
Patch 73J186 : Patch for the eMate 300
Patch 74J185 : Patch for the German MessagePad 2100

And finally, a Patch Remover which removes all user applied patches on NewtonOS 2.1 devices (you should never need this, but I supplied it just in case).

All files are as usual on SourceForge, who have been busy with the download interface, but it’s still not quite there yet, the patches are under the “Patches” section on the 40Hz download page .

Updates coming

Tue, 21 Jul 2009 00:00:00 +0000

This is just a short headsup, I’ll be making some changes to the Y2010 patch related web pages, and as usual, those changed pages will show up in the RSS feed.

eMate in 2010

Sun, 12 Jul 2009 00:00:00 +0000

The Y2010 patch for the eMate is now also almost ready, like the patch for the german MP2100, it still needs some testing, but things are looking good. The eMate seems to take the ROM board changes (which are necessary to erase a faulty patch) less lightly than the MessagePads. I had a couple of moments where I thought I had fried the little green machine, but it always recovered after a while. I’ll hopefully be able to release the patches during next week, still in time for 2010 ;)

Willkommen zur Zukunft, MessagePad!

Tue, 07 Jul 2009 00:00:00 +0000

I’m just putting the final touches on a fix for the Y2010 bug for german MP2100s. It seems a bit easier and more structured when doing it now for the second time :) So far no ROM board swapping was necessary, and some parts of the patch creation are now just copy and paste of existing code. It’s still exiting to see the new OS version “74J185” shown in the splash screen! Next up would be the eMate patch. Creating a patch takes about two days, I hope to find that time soon.

On another note, SourceForge has at least now fixed the download page so that all packages are listed, but direct linking to files just for a specific package is still not possible…

Damn you, SourceForge!

Sat, 04 Jul 2009 00:00:00 +0000

It seems that the new fancy UI on SourceForge has screwed up all external download links, and what’s even worse, it doesn’t show the complete list of packages for the 40Hz project anymore. That makes downloading files now impossible, I’m looking into this, but it should ideally be fixed on the SourceForge end…

RDCL on Windows

Tue, 30 Jun 2009 00:00:00 +0000

I just tried out the latest one click installer for Ruby 1.9, and it seems that the RDCL works out of the box :) I am using a generic USB to serial converter with the common Prolific PL 2303 chip, and after correctly setting the parameters, I could connect to my Newton without problems.

It’s great to see that Ruby 1.9 is slowly becoming more mainstream, and that installation is nowadays quite easy!

Patch 710031 and 71J059

Tue, 26 May 2009 00:00:00 +0000

One quick but important note: The 71J059 patch is not compatible with patch 710031 which Paul Guyot has created. I’m not sure about the reason, but before applying the 71J059 patch, the 710031 patch needs to be removed with the 717260 Install Override patch (it’s contained in the 71J059 patch ZIP file).

The Patch is out

Sun, 24 May 2009 00:00:00 +0000

I released the [Y2010 patch ](/Pages/Patch 71J059) for US MP2100 models yesterday, and hope to be able to work on the German model, the MP2000 and the eMate next. That requires reverse engineering the respective base batches for those models, and I decided to automate this a little bit. I started adding some functionality to the RDCL to read package files. If you have followed the RDCL development recently, you might have noticed that I also started putting together a simple Qt-based wrapper application for the nwt command line tool, to make operations like installing or syncing easier.

Patch Preparations

Sat, 23 May 2009 00:00:00 +0000

If you have subscribed to the RSS feed for Mottek, you might have noticed that not only these posting appear, but also all other changes I do to the 40Hz site. Makes it easier to follow what I’m up to :) I’m mentioning this because I’m about to release the patch for the year 2010 problem for the Newton. So don’t be confused if you see items about the related web pages appear in the news feed.

RDCL to the rescue

Wed, 22 Apr 2009 00:00:00 +0000

I recently got a new Mac mini to start playing around with Leopard-only software like MacRuby , and moved most of my day-to-day files and programs. The problem is of course that Leopard does not come with Classic, and apps like NCU won’t run.

Fortunately, the RDCL library and tools work quite nicely, and after downloading Ruby 1.9, I was able to get first my address book in vCard format onto the Mac (using IC/VC), as well as some other important notes. There’s still lots to be done in terms of converting data (ink and drawings would be great), but it’s a good start!

Blast from the past

Fri, 20 Mar 2009 00:00:00 +0000

I added now also the Mottek archive from 2003 onwards to the MoinMoin setup, so happy digging into really old stuff ;) I should soon be able to get back to the year 2010 fix .

If you can read this ...

Tue, 17 Mar 2009 00:00:00 +0000

in your feed reader, then also the migration of the Mottek blog to MoinMoin has worked. Just as a reminder, the RSS feed now lives at http://40hz.org/Pages/40Hz?action=rss&unique=1, but the old links should still work as well. I’ll convert the old blog entries prior to 2005 very soon as well.

Work on the Newton’s Y2010 fix has not stalled, but I want to make sure that I put in all the necessary safeguards to not cause more problems than the fix would solve. The main problem is that there’s no way to unpatch a Newton unless you have a ROM board for a different language, or an eMate’s ROM board.

Moin, Moin

Mon, 16 Mar 2009 00:00:00 +0000

I’m in the process of changing the backed of 40Hz.org from MediaWiki to MoinMoin to be able combine the web pages and the blog. Some hiccups are to be expected, so please bear with me :) I’ll try to put redirects into place so that external links still work.

John Arkley was wrong

Sat, 28 Feb 2009 00:00:00 +0000

Just to make it clear: Despite reports to the contrary, it is possible to create NewtonOS patches. It’s not for the faint of heart, and you need a german and a US ROM board to erase broken patches, but statements like “Apple is the only one who has the tools to build a new System Update,” and “Building and testing a System Update is complex and expensive process and no single engineer could do it” are plain wrong. The proof is on SourceForge for a simple test patch I created as the start for fixing the [2010 problem](/Pages/Newton Year 2010 Problem), and on UNNA for a more elaborate patch Paul Guyot has put together.

The real challenge is not so much creating a patch, but knowing what to patch and how.

Patch day

Mon, 23 Feb 2009 00:00:00 +0000

The Newton’s [Y2010 problem ](/Pages/Newton Year 2010 Problem) would best be solved with a low level patch which kicks in very early in the boot process. I have played around now for a while with the patching mechanisms, and got a nice pile of information gathered. I got a great start with the work done by Kip (Jonathan Knight), who reverse engineered the original 717260 patch by Apple and uploaded it to UNNA.

I created some test patches, and while it is a bit complicated, I was able to add new code to an existing patch, patch previously unpatched functions, remove patches completely, and change existing NewtonScript functions. In the process, I managed to disable one of my Newtons (I patched a function in the startup phase), but that should be reversible by changing the ROM temporarily, that will erase all patches.

Getting done even more

Mon, 09 Feb 2009 00:00:00 +0000

I’ve been using Nitch now for over a year, and it seems that it allows me to hit the right balance between planning and executing, thanks to a large part to the built in Notes and Todo applications.

I just added functionality to support planning still a bit better via the addition of “Someday” and “Waiting for” lists. I’m always amazed how fast it is possible to add functionality to Newton apps, and even though I’m sometimes worried whether the result is too much of a hack, I find myself still be able to maintain my creations after longer pauses!

I also improved some minor details in RDCL, mostly in the synching area. Ruby is quite hack-friendly as well!

My Email Habits

Mon, 27 Oct 2008 00:00:00 +0000

I think it is high time to apologize for my long delays in answering emails. They are in my inbox, and will remain there until answered, but I really have to put in more efforts to answer them faster. You’d wonder with all my attempts at GTD, I would be better in this area… At least, GTD seems to work as designed: Unless I create an “answering emails” project, nothing seems to move forward. So that’s exactly what I just did with Nitch. Let’s see how it goes!

Old Skool

Tue, 05 Aug 2008 00:00:00 +0000

My current way of working with the Newton to get things done is quite effective, but once in a while, I’m hitting a snag: Whenever I need to jot down larger amounts of data, brainstorm or scribble, my handwriting is getting in the way, as well as the limited size of the Newton’s screen size. In the GTD flow, the Newton is very well suited for the more structured phases like defining goals, next actions and projects, and marking actions as complete. But for free form brainstorming, I found it sometimes too limited.

In those circumstances, pen and paper seems to work better for me. This gives me also a nice excuse to shop around for productivity gadgetry :) I settled on a disc binder for the notes - for the pen, my initial choice was a nice classic Lamy 2000 ballpoint pen. But I always had some reservations regarding ballpoint pens, something just doesn’t feel right about them. Which means I’m back to fountain pens, and the world of ink stains ;)!

Goal achieved

Thu, 29 May 2008 00:00:00 +0000

Nitch is now at version 2.0.1 uploaded to SourceForge, and the latest major change was to add support for high level goals. I think I need to add some screenshots to the overview page here on 40hz to make things a bit clearer, but in brief, it is now possible to define roles (there is a FlashPoint-like way to switch to different views in Nitch via a folder tab), and add projects as goals for each role.

Planning goals for roles is usually an activity done at the beginning of the week, with daily adjustments if necessary. Nitch will prioritize actions higher when the associated project is a goal, and it will use a visual highlight for those actions.

I’ve been using Nitch now for about half a year, and it seems to really hit the sweet spot in terms of managing projects and actions, marking actions as complete, and getting feedback about achievements and priorities. All of that was tremendously helped by the NewtonOS - again, I have to say! It’s such a joy to program for it…

Habits and Goals

Fri, 16 May 2008 00:00:00 +0000

It seems that I’m joining the ranks of the many who want to combine GTD and the 7 Habits approach by Stephen Covey. When I first approached GTD, one of my concerns was which next action to pick, see for example my use of priorities in Flashpoint. This was however not the most critical point at that point, it was rather how to slice projects into small, actionable items to actually get something done.

Now that I’m getting better at creating next actions and projects (Nitch is working really well!), making the right choice among the actions comes back as an issue. For a while, I have been using ToDo items in DateMan as “themes of the day,” but this overlaps with my use of ToDo items for the hard landscape for that day.

Which brings me to the 7 habits. After re-reading Covey’s book, the approach to first define your roles, and then your weekly goals with daily adjustments makes an awful lot of sense. I’m still thinking whether goals can be expressed straight in terms of a set of GTD projects, but overall, integrating this approach into Nitch should be easily doable.

Ruby, Ruby, Ruby, Ruby

Wed, 30 Apr 2008 00:00:00 +0000

I’ve been working on and off with Ruby in the past, and really like the language. There are some quirks and inconsistencies, but overall, it’s still the best of the lot. I’m working with Mac OS X, Windows and Linux, and Ruby bridges the platforms quite nicely, including the user interface layer. Only threads and IO are a bit tricky.

The project I recently spent most time on is a framework for communicating with the Newton via the regular Dock protocol (it works over serial, Bluetooth or TCP/IP connections). It is in spirit a replacement of the Desktop Connection Library, written in Ruby. I’m using the 40Hz repository on SourceForge to host the Subversion repository in case you want to check it out. It’s nowhere usable yet though :) But I can connect cleanly to the Newton, and the next step is to fill in the missing commands to do something useful.

The purpose of such a framework is to help with the basic use cases of installing packages and dumping data from the Newton, but in a wider context, it would be nice to use it as the basis for synchronization or backup/restore.

Pen Mod

Tue, 29 Jan 2008 00:00:00 +0000

I noticed that writing on the Newton is easier with my rotring Quatto-Pen Data than with the regular stylus, but somehow, the rotring pen has a slight annoying rattle. On the other hand, we have tons of Pilot Super Grip mechanical pencils at work, which feel quite comfortable overall. Well, it seems that with a little bit of drilling and sawing and some super glue, it’s very easy to fit the “data” tips of the rotring (or other styluses) into the Pilot pen. And voila, the “Pilot Super Grip Data” is born :)

The reason I got stuck on this was that the Pile of Notes approach works very nicely, it just requires that you write down your thoughts the moment they come up.

The Quest for 'Done'

Sun, 30 Dec 2007 00:00:00 +0000

Here is one more round of updates to my “getting things done” quest: After rationalizing a bit more what is actually important in GTD for me, I came up with Nitch, a little tool which lets you use the Newton’s Notepad as a project repository where next actions are simply handled as checklist items. Check one item, and Nitch will show it as a next action, mark it as done in Nitch, and it will be removed from the project checklist.

I find it quite remarkable how in the end, any of my approaches to GTD is leading back to a very strict interpretation of David Allan’s book and articles. As soon as I deviate too much, things stop working for me. It is one of these situations where things seem obvious when reading them, but you have to come to the same conclusion by actively practicing the methods. As an example, Flashpoint did in the end not work well for me because it took too much time to create new projects. But having a complete list of projects is extremely important, otherwise you will always end up having open loops.

Pile of Notes

Thu, 15 Nov 2007 00:00:00 +0000

I’ll be changing my job soon (funnily enough, most of the things I do now will stay with me), and I need to plan quite a few things like scope, cooperation with others, team setup and so on. Early stages like this are quite interesting from the way they are organized: They look much fuzzier than day-to-day work, and there is a component which does not fit in very well with an approach like GTD, at least from my perspective. That component is brainstorming and information gathering.

I have used mind maps in earlier projects, and they did help, but also had a tended to have a more temporary character. The challenge with information gathering is that the brain does not like to be controlled in that area, and good ideas or important thoughts just pop up at any time - regardless whether they are connected to a current project or not. This is a constant process, and ideally dealt with at all times.

Recently I came across one method which was puzzling first, but does seem to make quite a lot of sense now to me: The Pile Of Index Cards. You really have to adopt the mindset fully, read the whole Wiki, look at the Flickr pages and let it sink in for a while - but then I got it:

Categorizing thoughts is difficult, so why not skip that step and just write down whatever comes up, and concentrate just on the thought, not the potential context
The context can vary, and it is established during the Task Force phase where all thoughts are gathered and evaluated
Thoughts come up all the time, and an index card seems just like the right size to gather one idea
Instead of trying to form and document a clear picture and set of thoughts, the focus is on non-discriminating information gathering. Maintaining a clear picture (revisiting, updating, categorizing, tagging, sorting) at all times is very difficult and time consuming.

The original POIC method uses index cards, but I think the Newton is actually equally usable, and maybe even better: You can carry all your thoughts with you at all the time, the POIC Dock is a bit to bulky for that. One drawback is however that you cannot spread out all the notes, look at them from a bird’s eye perspective, and cluster them. The Newton allows searching, and it would be possible to handle the Task Force phase by putting Notes which are connected into different folders, but it maybe still a bit cumbersome. There is nothing preventing us from printing all the Notes though, and handle this part not on the Newton but using plain old paper.

I will give this method a try, in the end it is pretty interesting to say “good bye order, welcome chaos” :)

Newton and GCC

Sat, 10 Nov 2007 00:00:00 +0000

My previous efforts of using GCC to compile C and C++ applications for the Newton are now documented a bit better here. I uploaded the tools I put together so far to SourceForge. They include some Lua scripts to turn ELF executables into NCT modules, and a template makefile with the proper compiler and linker flags. The tools should be cross platform, but a cross compiler for the ARM target is still required.

Using the tools should allow compilation of C programs and linking them with the Newton libraries. Using any of the C++ classes in the Newton libraries is however quite tricky, I don’t think it will work out of the box. The reason are the different calling conventions, name mangling schemes and constructor/destructor calls.

Distractions, distractions...

Sat, 20 Oct 2007 00:00:00 +0000

I wrote earlier a bit about Scrum and how it looks similar to GTD. The practical reason for looking at Scrum was that I’m working on a small project which uses Scrum principles (not the full blown version). As it seems, most of the tools available to support scrumming are a bit too heavy for my needs, so I took the opportunity to write my own small app - and learn Ruby on Rails in the process :)

The result is now on SourceForge: A small RoR app called BagLock. Not perfect, but maybe somebody finds it useful.

GTD again... and a bit of Scrum

Sun, 16 Sep 2007 00:00:00 +0000

I finally found some time to document my various GTD setups. It is quite interesting how much time it takes to realize what is actually behind GTD, and what the real issues in doing one’s work are! Another interesting dimension is the connection between GTD and the Scrum management method (it’s not only usable for software development). To me, projects look awfully like user stories, and sprint backlog items like next actions, just on a slightly bigger scale. And the principle to frequently identify those things which would bring you closer to the “done” state, and start working on them instead of trying to come up with a project and requirements plan at the very beginning looks very much like the GTD process to me.

Blunt 2 Status

Sun, 26 Aug 2007 00:00:00 +0000

After fixing the MySQL issues on 40hz.org (seems that some of the tables got corrupted), here is a status update for Blunt 2…

Overall, I’ve made some good progress on Blunt 2 over the summer. Right now, Blunt 2 is almost usable for connections initiated from the Newton side. I rolled in some stability improvements, changes to the service discovery, better preferences and more diagnostics code. There are still some stability issues which I suspect are coming from the interrupt handling routines, and some issues regarding automatic startup and shutdown of Blunt 2. Also, accepting connections is not implemented at all on the NewtonScript level. Still, I’ve been able to transfer data using IC/VC and Neo, which is right now the most important use case for me.

One interesting side project was to improve logging facilities, and I’ve settled now for a very simple but effective mechanism: I added a logger class which writes data straight onto the serial port at 9600 bps, which means a simple “cat /dev/…” command is enough to get the log outout. No more messing with Hammer :)

The current snapshop is as usual on SourceForge in the Blunt 2 release area.

PS: There were also some minor changes to IC/VC regarding export of names with multiple addresses which was problematic before with Mac OS X Address book.

A little hammered

Fri, 20 Jul 2007 00:00:00 +0000

The Newton, that is :) I haven’t gotten the complete Newton debug protocol put together yet, but I have at least a way to connect and get some logging done. Since I haven’t used Hammer to actually debug much (Blunt for example is very asynchronous which is not easy to debug anyway), this is enough to get rid of Hammer for regular development work. The problem with Hammer is that it loads the CPU 100% when connected, and it has a tendency to hang and hog the serial port. And it runs in Classic only.

There is still lots missing from the actual debug protocol, right now especially initiating and terminating debug sessions is not done properly since it involves not actual commands, but rather setting memory values remotely on the Newton. Reverse engineering those parts is however lower priority now because I really want to get Blunt 2 done!

At least I tried

Mon, 02 Jul 2007 00:00:00 +0000

So, it is possible to get CFront to work. I downloaded the 3.0.3 release, massaged the source a bit and compiled it on MacOS X. The output looks very close to what the Norcroft compiler generates, but it’s not close enough. And the reason is simple: The Norcroft compiler in the Newton C++ Tools is based on the 2.1 CFront release. Too bad, there is no source available for that. I guess this does not qualify as fun for many people, but maybe I get a degree in software archeology at some point… we’re talking about code which is 15-20 years old. No further comments about the quality of the CFront sources, they compiled after all, but looking at them is as much fun as poking a fork into your eye.

As a bit of background information, the main issue with C++ on the Newton is the incompatible vtable format: The Norcroft compiler does not handle multiple inheritance well as far as I can see, so it uses a quite simple vtable format, and all later compilers changed that somehow (either by using thunks or stubs, or by adding offset fields). Unfortunately, the vtable format has a direct influence on the generated code, calling virtual functions looks quite different from compiler to compiler. In addition to that, the vtables in the Newton ROM are actual jumps, whereas other compilers just use function addresses.

What will work though is to use C++ without virtual functions, and to stick with C.

C++ ruins it all

Thu, 28 Jun 2007 00:00:00 +0000

Development tools for the Newton are becoming more and more an issue - especially the C++ part of that problem. The NewtonScript side looks actually quite ok now with DyneTK, but we are still stuck with the ancient Norcroft C++ compiler for classic Mac OS (the 68k variant). Not that it won’t work, but it makes moving to an Intel based Mac quite hard. Add to that the problem that serial ports don’t seem to work here under BasiliskII and Mac OS X, it seems that you need a PowerPC or even 68k based Mac for any Newton C++ development.

I dug a bit deeper recently, but just to confirm that no version of GCC will ever be usable for C++ programming for the Newton. C will work, but the C++ code generated by GCC and the code in the Newton ROM won’t play nicely due to vtable issues (among other things).

Well, one last straw still exists, although it is actually slightly insane to even consider this option: The Norcroft compiler is based on CFront, so theoretically, you could compile CFront and use it to convert C++ code into C code, which could then be compiled by e.g. GCC for the Newton. The old CFront sources are available, but a quick attempt at compiling them resulted in a bunch of errors… maybe it needs a bit more than just a quick attempt.

iPhone

Tue, 23 Jan 2007 00:00:00 +0000

There has been lots of discussion about the iPhone in many places, among them of course NewtonTalk, and Leander Kahney dropped me and others a quick emailasking about my option on it, especially how it relates to the Newton… well, I said it before and I’ll say it again :) - the Newton is hard to replace since it is so reliable and versatile. There’s no way I would trust a device without a user replaceable battery with my data, or use a device which needs to be recharged frequently without any usable emergency power source like AA batteries. A PDA simply has to work, otherwise you’d worry more about the thing doing its job than actually being productive with it. After thinking a lot about what I need for both organizing both my personal and work life, I concluded that it’s either the Newton … or paper :)

In case you wondered ...

Mon, 04 Dec 2006 00:00:00 +0000

… why I haven’t blogged much recently, and what happened to Blunt, no worries. I was (and am still) working on something else for my trusted Newt :) I should have something ready to be released soon, and hope that people will find it useful!

The Warnings...

Mon, 09 Oct 2006 00:00:00 +0000

… in the header file used to interface with the serial chip seem to be quite appropriate. It is not really rocket science to send and receive data, but it is pretty difficult to pinpoint any problems because they are usually timing related. I made some changes in the sending part which enabled more stable OBEX operations with Blunt 2, but it looks still a bit incomplete - sending data in large batches results in errors on the receiving end, which hints at some data corruption.

Getting closer...

Sun, 17 Sep 2006 00:00:00 +0000

I just used Blunt 2 for first time together with Neo and IC/VC, successfully transferring data from the Newton to a PC over OBEX. It seems that there are only two things open to enable basic functionality: instability caused by unclear memory management when handling Newton events, and flow control for sending and receiving. Neither is impossible to resolve :)

Overall, this was a quite important milestone!

Messy

Sun, 03 Sep 2006 00:00:00 +0000

I’m still trying to solve the flow control problem with Blunt 2. I’m using a Linux PC to send data at the highest possible speed over RFCOMM, and without any handling besides the raw BT stack, it sort of works, but it is buggy and crashes the Newton. The code which is necessary to make it stop crashing, as well as the code needed on the NewtonScript side to actually do something useful slow things down, causing buffer overflows. The flow control code which I put in didn’t really work, but I’ll still see if the more advanced credit based control would work better.

On the bright side, it seems that when using MacOS X, the speed is just about right for things to flow smoothly - about 5kByte/sec, including handling on the NewtonScript side (I’m storing the data in a 1k binary object). That means that I only need to add the sending part, and the scenario where the Newton initiates the connection should be complete - sending and receiving should work, with much improved error recovery. No more -18000 and -19000 errors :)

10 to 15 ...

Sat, 26 Aug 2006 00:00:00 +0000

… is the raw transfer speed in kilobytes per second which Blunt 2 manages. This is however not taking any overhead into account which is still missing, most notably sending the data from the Blunt stack to the CommTool, and then using that data in NewtonScript. It’s very hard to guess how much the overhead actually is, but the above number is quite good. I also managed to solve some stability issues with the stack, the only two remaining stability issues are the potential paging out of the interrupt handler code, and the implementation of the CommTool.

Flow control...

Mon, 21 Aug 2006 00:00:00 +0000

… seems to be something I need to put in place next in Blunt 2. Just for fun, I wanted to test the raw transfer speed over Bluetooth, and was spending quite a lot of time trying to figure out why I get corrupted data after a certain time. Turns out that the receive buffers on the PICO card are too small, and with all the logging, Blunt 2 is reading too slowly from the card, resulting in an overflow. The good thing is that with timers being now much easier to implement in Blunt 2, adding flow control should be not too hard. Ideally, this would be credit based Bluetooth 1.1 flow control, but it looks like that the Mac OS X stack only does the serial port emulation of the 1.0B version, which is a little less elegant.

Testing, 1, 2, 3, Testing, ...

Wed, 02 Aug 2006 00:00:00 +0000

Most of the pieces in Blunt 2 for sending data from the Newton seem to be in place - I can successfully discover devices, pair, get services and connect using a regular NewtonScript endpoint. This last part is actually pretty important. There are however still some quirks, the most annoying issue is stability. I think that both the server and the CommTool have problems in this area. The server probably because it is using the TSerialChip directly, and there are lots of funny things going on with DMA and interrupts, and the CommTool looks problematic because it is tied very closely to the NewtonScript world and its event loop.

Still, I think this will not be magic to sort out, and in the meantime, I’ll try to get an idea what the performance of Blunt 2 will look like. One promising thing is that the always on and multiple simultaneous connection features look feasible. And that opens the door to fancy things like browsing your Newton from a desktop PC via OBEX, mapping for example Names to vCard “files” which can simply be copied, or Notes to “files” with multiple representations (e.g. “Note.nwt”, “Note.xml”, “Note.txt”).

PS: Native Bluetooth support in Windows XP is pathetic, third party software like Widcomm is unstable and Mac OS X is so much better - there, I just had to say that after a day of fighting with Windows ;)

Wiki

Sun, 16 Jul 2006 00:00:00 +0000

I just moved to MediaWiki to maintain 40hz.org, it makes updates much simpler - hopefully that will encourage me to update information more frequently :) There is still one issue regarding uploading packages, source code and other data (the Wiki is not really made for that), one option is to use SourceForge exclusively as the data repository. That would have the advantage of having only one location for multiple releases, but the disadvantage is that it is a bit more complicated to download things. Also, direct download of packages would not work.

I will gradually move most of the pages on 40hz.org to the Wiki, but for now, pages besides the main page are still “regular” pages. I also disabled editing by non-registered users, and since I’m the only registered user, that make things simple :) But it might make sense to allow others to edit pages in the Wiki, let me know if you have specific ideas. There is of course always the WikiWikiNewt on UNNA.

Blunt 2 Status

Wed, 28 Jun 2006 00:00:00 +0000

In case people are interested, here’s where Blunt 2 is standing at the moment: Bluetooth Setup looks quite ok, but getting Service information from remote devices is not implemented. Im testing with my Pico card, which hopefully helps people who are using the same card (it’s a cheap and readily available choice). I sent a version earlier to Daniel Padilla, unfortunately it just crashed his machine. My current development machine has been brain wiped, so it should be in a state similar to other Newtons, let’s see if I can prepare another version for testing. Ideally, people who want to test have Hammer installed so that we get a bit of logging data.

There are some open issues regarding the initialization of the components of the Bluetooth stack, i.e. how the HCI, L2CAP and RFCOMM handlers are started up. It’s a bit hackish at the moment, starting from the bottom of the stack and moving upwards. Maybe initializing the high level layers first is cleaner.

Brain Transplant

Sun, 04 Jun 2006 00:00:00 +0000

One problem I have sine I bought my first Newton back in 1995 is that my handwriting is bad, in fact I have trouble reading it myself ;) However, Newton’s printed recognizer is actually doing a quite good job. To improve recognition, I use alt.rec…, and since most of the things I write in English, I finally decided to do a brain transplant, moving the US ROM from one of my spare Newtons to my main Newton. The physical process is very simple, but moving data back and forth required a bit more planning. I wanted to take advantage of the fact that the memory gets wiped completely after a ROM transplant and decided to see if I can preserve all my data without a backup. For that to work properly, I moved all packages to my memory card, and then used Soup Mover to move the data as well. Settings are lost, but they are usually not that crucial. Everything worked fine in the end :)

Connecting - Disconnecting - Connecting ...

Thu, 06 Apr 2006 00:00:00 +0000

The overall design and implementation of the Blunt server seems to work ok, I can connect from other devices for pairing and getting Bluetooth services, repeatedly opening and closing connections. This is actually the biggest difference to the current Blunt implementation. Now it should be possible to have services running permanently, and not only when a component uses Bluetooth. Bluetooth Setup is also now starting to get more and more adapted to the Blunt server, which makes discovery problems hopefully a think of the past (the discovery time will be configurable as well). One important thing to sort out still is memory consumption, I think I need to add malloc/free tracing macros to the code.

Down to the metal

Thu, 16 Mar 2006 00:00:00 +0000

Another solution to the GCC problems for Newton development is to use Assembler. Sounds crazy? Maybe, but that’s how I started programming ages ago, and at least for low level stuff like Blunt (which is mostly about moving bytes around), it might actually be quite suitable. Only interfacing with the NewtonOS and C++ requires some effort, but there’s no reason not to use clever preprocessing or macros to simplify it… this is of course no cure to port existing C or C++ code.

GCC again

Mon, 06 Mar 2006 00:00:00 +0000

Now that the new Intel Mac mini is out, it would be nice to upgrade to it at some point, but the question of the old Newton development tools comes up again… I guess there is really no way around replacing the tools with GCC and NEWT/0. For GCC, the problem is that a new backend is needed to take care of the proper creation of the vtables, and last time I looked, NEWT/0 does not create NSOF output, but rather NewtonScript bytecode for functions. Well, I guess neither problem requires rocket science to be solved, just some work :)

Putting things together

Tue, 14 Feb 2006 00:00:00 +0000

It looks like the architecture of Blunt 2 is now solid enough to bring in the missing functionality one piece at a time. Currently, my goal is to adapt Bluetooth Setup to use Blunt 2, and at least the lower layers are already in place. Driver selection, resetting the module and discovery are working. Lots of multitasking happening, and it’s nice to be able to use more and more features of the actual NewtonOS like message sending and tasks!

Next Action...

Tue, 10 Jan 2006 00:00:00 +0000

I’ve been doodling around with implementing some stuff from “Getting Things Done” on my Newton, and it seems to work reasonably well (I was getting too annoyed by abusing my Inbox as a to-do list and working on stuff as it comes up instead of working on stuff that’s actually important). I’ll probably write a bit more about this later, right now, the biggest problem seems to be my handwriting ;). I implemented some stationery, and I’ll probably add some more hacks to the Notes application, as well as integrate a bit better with DateMan. All that will go of course to 40hz.org when it’s ready. And then, I’m going to get back to Blunt 2, promise!

PCMCIA cards

Wed, 19 Oct 2005 00:00:00 +0000

While the internal Bluetooth module in my development Newton is sick with some unknown hardware problem (it doesn’t send, maybe the 3.3V line got loose), I spent some time to see what the problem with the PICO card and Blunt 2 was. Seems like things are actually quite ok, the card uses regular byte I/O, not DMA, and there were just some bugs in the code. The only irritating thing now is that as soon as I try to send some L2CAP packets, it returns a hardware error. Well, time to restructure the code so that I’m able to have a bit more control over timing, maybe add some delays here and there…

Wedgie

Tue, 11 Oct 2005 00:00:00 +0000

Ok, easy solution to the page fault in the interrupt handler: I wedge the interrupt handler code between two functions which will definitely be called before any interrupt occurs. The code is quite small, this is however something to still watch later. At least the symptoms are very easy to interpret, a “Deep Toast Alert” doesn’t happen too often :)

Something Interesting...

Sun, 09 Oct 2005 00:00:00 +0000

… again ;) - I’m currently trying to figure our why the interaction with the TSerialChip crashes when talking to a PCMCIA card as opposed to the internal Bluetooth module. Seems that it’s not the actual interaction with the chip (the functions in the TSerialChip16450 class are quite simple), but it’s a general problem with the interrupts they cause. Something that could as well happen with the internal module. Initializing the chip generates interrupts pretty much right away, and the problem is that accessing my interrupt handling code causes a page fault, probably because it hasn’t been loaded yet. An easy way around that would be to force it to load by referring to it explicitly, but that’s not terribly elegant. I think that Paul has hit the same problem in the ATA driver much earlier than I did, so maybe I’ll check if I can use his workarounds as well. One problem might be however that Blunt 2 is not produced using Packer, instead it’s a regular Newton Toolkit project.

Slowly Progressing

Tue, 06 Sep 2005 00:00:00 +0000

Much of the Blunt code can be now reused, but there are some bigger parts which need to be redesigned completely, e.g. to support multiple active RFCOMM or L2CAP connections. In the end, it should pay off though, having an always-on Bluetooth stack is very handy, especially in the way I use my Newton: during the day it’s always on and running off the AC adaptor, otherwise I’m using rechargeable batteries.

Under the hood, things are looking quite interesting. Working with tasks, ports and messages seems easier than I thought, and the only thing which is a bit open at the moment is memory management. It’s possible to access data across task boundaries, but maybe shared memory would be the cleaner way to do things.

Serial Problems

Sun, 31 Jul 2005 00:00:00 +0000

Blunt 2 is supposed to be implemented as a standalone task which talks to the serial chip directly. It seems that the warnings in the DDK header files about using TSerialChip are justified… interfacing with the 16450 chip which is usually used for PCMCIA cards fails right in the initialization with a “deep toast” alert. I guess there is some magic happening with the interrupts, supervisor mode and other low level stuff.

The good news (at least for me) is that it works when talking to the Voyager chipset which is in charge of the internal serial port. That means that my internal Bluetooth modules should work fine. The only requirement is that instead of pain old serial communication with the chip, DMA has to be used. Another fun thing to reverse engineer! But it looks ok now, and besides getting some rudimentary Bluetooth stack up and running, I ‘just’ have to implement the CommTool counterpart which sits between the NewtonScript world and the Bluetooth stack.

Classic - here to stay?

Tue, 19 Jul 2005 00:00:00 +0000

Finally I found some time again to fiddle around with cross compiling for the Newton… my Cube needed hardware and software upgrades, but Tiger running on a dual 1.2GHz G4 is quite nice. The bad news is that GCC in the versions I tried it (2.95.3 and 3.3) generates or assumes virtual tables which are not compatible with what the original NCT compiler generated. This restricts any usage to either plain C code, or classes without virtual functions. In addition to that, the generated code didn’t look particularly efficient, it is also about 50% larger than the Norcroft code.

Well, thanks to Paul’s toolserver, I can at least use Xcode as the development environment. The main problem with Classic is actually not using MPW, but Hammer, which sometimes hangs or crashes the complete Classic process… an RDI compatible replacement for Hammer would definitely be great!

To the Moon

Sat, 02 Apr 2005 00:00:00 +0000

One clearly underestimated programming language is Lua. I’m using it now to transform the ELF output of GCC to a Newton NSOF file. It’s amazing how flexible Lua is, and it would definitely be my first choice for any kind of system to replace the Newton. And it looks at least a bit like NewtonScript. The only thing missing are symbols, but I guess that adding them would mean adding a whole new paradigm or way of programming to Lua. This is where the less flexible NewtonScript is still ahead of most mainstream programming languages.

Distractions

Sun, 27 Mar 2005 00:00:00 +0000

Well, here we go again… instead of working on one thing, something else gets in the way and is way to interesting to not play around with. In this case it all started with the ancient Norcroft compiler crashing and taking MPW with it, just because I created some more complex class structures. So why not use something else instead? Xcode is a half decent IDE, there is a readymade ARM cross compiler for Mac OS X, and the NTK and NCT formats are also all available, e.g. on UNNA. Currently, there are only two things missing for bringing C++ programming for the Newton to this millenium: getting the runtime functions into a stub library, and duming the ELF image the GNU compiler produces into an NSOF formatted file. The runtime library is in principle easy to build, it just requires mapping the function names from the GCC mangled format to the Norcroft format which can be extracted from the debugger images. I guess there’s a way to do that automatically, but for now, I just add each undefined external function manually to the stub library. Some other smaller issues are exception handling and constructors, but I’ll get to that once the first module loads properly.

Moved to WordPress

Sat, 26 Mar 2005 00:00:00 +0000

Hurricane Electric moved my account to a new machine, and it seems that there’s some incompatibility with the Berkeley DB the Mottek data was stored in on the old machine. Well, I don’t really feel like setting up Movable Type again with all that registering stuff now requried, so I just switched to WordPress. Seems to be quite painless so far. The old pages are still available at http://40hz.org/mottek-old/.

0.7.6

Sun, 04 Jul 2004 00:00:00 +0000

Finally. I think I nailed the bug in Blunt which caused the random restarts. Or rather, I replaced the code which I suspected to be not working. The root cause was clear, data being received with no place to store. In the original TAsyncSerTool, the data is left in the input buffer, and before, I was removing it and storing it in a temporary buffer in Blunt. Now I just check if a HCI packet is complete, and if not, I just leave it in the input buffer. Seems to work quite nicely, so everybody having trouble with Blunt causing restarts should switch to 0.7.6!

Poll? Final?

Tue, 27 Apr 2004 00:00:00 +0000

The T68i is now accepting data over BtOBEX. It does some things differently in the RFCOMM connection establishment, mostly regarding compatibility between the 1.0b and 1.1 versions of the Bluetooth spec and the use of the Poll/Final bit. Also, there was a small bug in Blunt. I’ll have to test my changes with some other devices and also GPRS, but the T68i looks promising at the moment.

In the ER...

Mon, 26 Apr 2004 00:00:00 +0000

… are currently an older BT2000E Bluetooth card and a SonyEricsson T68i. Thanks go to Ronnie and Marty for the loaners! A quick examination showed that the BT2000E is probably a tough case (the serial chip is using some unknown parameters), but the T68i looks better (it has some problems on the highest RFCOMM layer). More soon.

Serial Drivers

Tue, 16 Mar 2004 00:00:00 +0000

Now some background information about making the Air2Net card work with Blunt… The card uses a 16950 serial chip, which is in the default configuration compatible to the good old 16450 chip. The problem is however that this also means that the transfer buffers are only one byte “deep.” This causes immediate overrun errors as soon as a bit more traffic is going over the wires. Fortunately, the ‘950 is simple to switch into other modes (‘550, ‘650 and ‘750), so all that was necessary was to use the TSerialChip16450 class to talk to the chip and set some registers. The result is a new simple helper class which will from now on contain the chipset specific parts of Blunt.

On a related note, Daniel Padilla has gotten his hands on a Conceptronic CF card. Unfortunately, connecting to that card crashes the Newton. It too uses a 16950 chip, but it seems that it advertises its I/O space in a way that confuses the Newton, resulting in a crash when the registers are accessed. It would probably be possible to hardwire the correct parameters, but Daniel wants to take the card apart and use the internal module directly, similar to my setup.

Bluetooth Hardware

Tue, 02 Mar 2004 00:00:00 +0000

I’m currently working on an AmbiCom Air2Net BT2000 driver for Blunt (thanks for the loaner, Zac!). It is highly likely that this CompactFlash card will be supported without restrictions. So we have the PICO Card, the internal modules, and now the BT2000 card. The BT2000 card is not very expensive and consumes very little power. It looks like a good choice if you are living in the US since it is readily available there. For Europe, the PICO cards show up regularily on eBay, but I suspect that the Conceptronic Bluetooth CF cards could work as well.

Field testing

Fri, 20 Feb 2004 00:00:00 +0000

The internal Bluetooth modules behave really well. No ugly cards sticking out, very low power consumption and adequate speed. Some improvements would be automatic reception of items in the Inbox, which should not affect battery life at all. Also, there are some sporadic restarts which are probably due to a buffer overflow in the NewtonScript adaption layer. Otherwise, I’d say that this was definitely a very successful hardware and software hack!

Case closed

Thu, 12 Feb 2004 00:00:00 +0000

My Newton’s case that is. It seems that the current wiring is more or less ok so I screwed everything back together. The Bluetooth module is powered when the Newton is on, and it’s off otherwise. I still needed to figure out a reset method for the module since the two power lines are independent. There is this practical serial channel 3 select signal on the internal J6 connector which works nicely as a reset line. Now whenever the Newton reboots or is powered up, I reset the module. Power consumption is great, when active, the total is always under 100mA.

An alternative to this would be to power the module all the time and use a voltage regulator to get the 1.8V out of the permantently on 3.3V line. Maybe another day :)

But there is still a Heisenbug in the whole setup. I’m not exactly sure if the channel 3 is up to speed or if I really should get the RTS/CTS handshake working, for now I have to use a Wait(1) after each HCI packet. That’s a minimal overhead. Would be nice though to know what’s going wrong.

Much better

Wed, 04 Feb 2004 00:00:00 +0000

The switched 3.3V line works. I hope it doesn’t cause any long term problems. Now the module is powered whenever the Newton is switched on. It takes under 10ms for the module to get into stable state, so switching it on and off is not really a problem.

Somebody with the skills to solder the dongle killer should be able to wire the BT module as well. Of course, a ready made module like the SER-001 would be much nicer!

Power trip

Tue, 03 Feb 2004 00:00:00 +0000

The problem of the module sucking too much power when the Newton is switched off turned out to be quite interesting. The 1.8V power used for the module core is turned off when the Newton is off. But the 3.3V for the I/O lines and the memory is still on. This causes the module to go into some weird state where it drains about 40-50mA. Now I’ve got to either find a 3.3V source which is also switched off, or a 1.8V source which is constantly on. There is already a candidate for the switched 3.3V source: The power supply for the PCMCIA slots…

This sucks ...

Mon, 02 Feb 2004 00:00:00 +0000

… a bit too much power. The Bluetooth module is supposed to draw only 12mW, but it drained my (admittedly very worn out) rechargable battery pack in 10 hours. And this while the Newton is off. Frank Gruendel has measured that the Newton draws about 1mA when off, which probably comes to around 5mW. Clearly, something is wrong. But after Reading The Fine Manual - which always seems to be a good idea, if you find it in the packing material -, I noticed that the RX and CTS lines need a 1k resistor. I’m now letting one Newton run down it’s battery while switched on, and another one switched off but with the module sans RX and CTS lines installed. This should give me an idea how much energy my battery packs still holds and how much the module will draw.

In the worst case, I’d have to add a switch. Ideally however, this problem (if it still exists after adding the resistors) should be solved by controlling the power to the module via the DTR line. Unfortunately, I don’t have neither the tools nor the skill to build something myself - it’s already quite astonishing that I got this far!

The final setup I’m planning to use also requires a bit of Newton case modding. The antenna should stick out of the case a bit because the case is shielded from the inside with some sort of metal layer. I scratched some of that off in an area where the module sits, and the range is actually quite good. The antenna is now there where the plastic cover for the SER-001 (or modem or audio-in or …) was. Not too attractive, but hidden behind the port door all the time. A bit epoxy modelling might take care of this later. For now I’m enjoying the first Newton with a built-in Bluetooth module, surfing the web wirelessly!

Yes, yes, yes!

Sat, 31 Jan 2004 00:00:00 +0000

Wasn’t this fun! I struggled quite a bit hooking up one of the internal Bluetooth modules to the Newton, but finally managed to get some results. Turns out that the CTS and RTS lines need to be switched and that it helps a lot if you actually provide all necessary power lines to the module. I was trying for a while to run it on 3.3V alone, but the actual core needs 1.8V as well. I’m sure that any hardware designer would have ran out of the room screaming observing how I found all requried connectors on the mainboard, but so what :) The thing runs and answers nicely to a HCI reset command with a command status event. Let’s hope that things stay that way!

They're here!

Thu, 29 Jan 2004 00:00:00 +0000

Two internal Taiyo Yuden Bluetooth modules. Now let’s see how they can be wired to the Newton’s internal serial slot.

No SyncML for you!

Wed, 21 Jan 2004 00:00:00 +0000

I played around a bit with iSync and mod_ssl to see what’s going on behind the scenes when using .Mac to sync your data between devices. Too bad things haven’t changed much from this. The protocol to the .Mac servers is clearly not SyncML (and probably will never be. Apple wants to make money with .Mac). That makes a universal SyncML client for the Newton a bit more limited in scope. But so what :) I think I’ll still give it a shot at some point!

Finally

Fri, 16 Jan 2004 00:00:00 +0000

Neo 0.8.5 contains an important bug fix that now enables sending OBEX data via Bluetooth to MacOS X and is required for correct OBEX transfers from the Newton in general. The bug was that the length of the overall object was sent incorrectly which lead to a timeout during transfer. Things look actually quite usable now!

Timers cont.

Sun, 11 Jan 2004 00:00:00 +0000

Timers are working now on the comm tool level. Most things are already in place, just a custom event class needed. A comm tool is running as a task and has already a port to where you can send messages. The message handling function is also implemented, it’s called HandleRequest. I now send deferred messages from Blunt to itself and have a nice timer mechanism. This helped in getting around the lost packets issue with MacOS X.

Windows

Thu, 08 Jan 2004 00:00:00 +0000

Ok, pairing from the Newton works now which is at this stage more or less required for Windows. But Windows doesn’t seem to like the duplicate L2CAP packets which Blunt sends for connection establishment. Seems like I have to figure out how to use timers in Blunt… Also, Windows refuses to accept OBEX data items. Could this be the same bug which causes Mac OS X to silently drop incoming objects?

Vacation is over

Wed, 07 Jan 2004 00:00:00 +0000

Time to start hammering again - and working off my backlog of emails! I just made some small adjustments to Blunt to improve Windows support. Pairing is the first step, and it looks like initiating pairing from the Newton works at least to Mac OS X and my 3650.

I also investigated SyncML a bit. I’m not sure if Apple will ever open iSync enough to allow true SyncML integration… they seem to use server authentication from iSync, most likely to see .Mac services. Instead of using iSync and .Mac, my plan is to play around with my own SyncML server instead and see how much effort a SyncML client for the Newton might be. I got Sync4J running on my Cube, and the vCard and iCal files used by SyncML should be importable into Address Book, iCal and also IC/VC.

What's next

Sun, 14 Dec 2003 00:00:00 +0000

Ok, the most obvious things to work on next would probably be the bugs and feature requests from Sourceforge. One thing which is however not there is to increase stability for Blunt. This covers two things: proper shutdown of the driver in error or disconnect cases to avoid the dreaded “communications port busy” error. And then safeguards to prevent crashes when talking to devices which I cannot test with. This concerns mostly Sony Ericsson devices which is actually a bit surprising. Ericsson has always been a good Bluetooth implementor, so I wonder if they interpret the standard a bit differently than I do. One thing they certainly can do is to be more agressive in their implementation since they know what should work.

Surfin'

Thu, 11 Dec 2003 00:00:00 +0000

I now have a PPP daemon running on my Cube, accepting incoming connections via Bluetooth. Works like a charm for mail and web surfing with my Newton, using Blunt. It’s almost like WiFi, only the power consumption is lower. The setup is quite simple: Add a new serial port on the Mac via the Bluetooth Serial Utility, add a Bluetooth internet setup with the correct name server and other parameters, enable IP forwarding with ipfw and natd and start pppd on the new serial port.

These are the commands I used to enable IP forwarding (en1 is my outgoing interface):

/usr/sbin/natd -dynamic -interface en1
/sbin/ipfw -f flush
/sbin/ipfw add divert natd all from any to any via en1
/sbin/ipfw add pass all from any to any
/sbin/route add default 10.0.1.1

I’m behind a firewall, so there are no other fancy filtering rules. The firewall is at 10.0.1.1, so that should be the default route to the outside world. The pppd command line looks like this (Bluetooth-Serial is the name I gave to my serial port):

/usr/sbin/pppd /dev/tty.Bluetooth-Serial 230400 noauth local passive proxyarp asyncmap 0 silent persist :192.168.1.110

When discovering services from the Newton, there are now two serial ports, and one of them is the new serial port. The other is the PDA sync port. Unfortunately, the exact port numbers seem to change, but it’s easy to find out using trial and error.

Bluntness

Mon, 08 Dec 2003 00:00:00 +0000

The latest changes to Blunt made the L2CAP connection establishment phase a bit more stable. It’s not really clean, but works: Instead of using a timeout and retransmit in case the connection establishment stalls, I just send the packets twice, no matter what. They’re marked using the same packet identifier so the peer can throw the second one away. Likewise, I ignore any duplicate response. A more difficult issue is connection closedown. It seems that proper disconnect at least in the RFCOMM layer is needed. Otherwise, peer devices don’t finish an OBEX transfer (or discard the transfer such as MacOS X). It’s a bit trickier to get into closedown since I’m not sure how much of the comm tool is still usable at that point. First attempts just caused a reboot.

Still open is also the SDP and some nasty crashes using the P800. I’m not sure why the P800 causes the crashes, but I’ll probably have to add some safeguards. The SDP is likely easier to fix. Some improvements for Windows support are also needed in that area.

Unrelated to that, adding gzip support to Courier was certainly a success. 8k/sec over GPRS are quite nice. Still, NHttpLib could do well with a rewrite…

Clock offset?

Thu, 27 Nov 2003 00:00:00 +0000

One theory for the missing Bluetooth packets on connection establishment is that I don’t send the clock offset when connecting. It doesn’t seem to matter on the 3650, but maybe the USB dongle on the Mac works better if the clock offset is available. It helps keeping packet transmission in sync… for now, connections to the USB dongle succeed only the second time around, and my guess is that the clock offset is at that point known.

NIE Support

Wed, 19 Nov 2003 00:00:00 +0000

After exploring the various bits and pieces which NIE installs, I managed to put together a first version of a “NIE Nitro & Blunt” module. It reuses much of the Serial & Modem module and adds a new internet setup (IrDA for now, Bluetooth will follow). This will make the patched NIE module (which has to be currently used with Nitro) obsolete. One open issue is the connection slip for Blunt since that requires a field for the peer device, but I don’t expect that to be too complicated. Getting closer to true Bluetooth integration…

More Bluetooth Setup

Sun, 16 Nov 2003 00:00:00 +0000

Discovery and Pairing look ok. There is still the issue with crashes after sending data (or a -48200 error), and Neo should not register transports for IrDA and Bluetooth when they are not installed. But the Bluetooth routing slip now uses the discovered device information and sets the pairing link key if available. The current archive on the Experiments page contains these latest developments.

The tricky part is now to get the same level of integration for Internet Setup… and not to forget about a more intelligent SDP server implementation.

Back to SDP

Wed, 12 Nov 2003 00:00:00 +0000

I made some progress with the Bluetooth Setup application (it’s part of the Blunt.sit archive on the Experiments page). Discovery works, but overall, it’s still quite rough. The preferences for chosing the Bluetooth device settings are integrated as well and are fetched by the comm tool when an endpoint is created.

The next steps would be pairing and discovering remote services. This probably one process, i.e. pairing also gets the available services on the peer device. This requires some additions to the SDP layer of Blunt. Not my favorite part…

The last steps are to use the discovery and SDP information in Neo to build the route slip when sending data to another device. I also have a vague idea on how to do the same for Internet Setup, making it possible to chose a peer device for a PPP connection. If my idea works, it will help Nitro as well and make the patched NIE Serial module obsolete.

Much better

Tue, 04 Nov 2003 00:00:00 +0000

The Blunt PPP speed problem was indeed caused by the wrong MTU settings. While trying to fix a problem when sending data via OBEX, I saw that the code for correct data segmentation during sending was still missing. The workaround at the moment is to use a small MTU size which will fit into the HCI buffer to the card and maybe add segmentation later. PPP over GPRS works fine now… the Newton is not a speed daemon, but it is definitely enough for browsing the web with Courier. And there is still the option to add gzip compression to NHttpLib.

Tricky, tricky...

Mon, 03 Nov 2003 00:00:00 +0000

The interactions between the various buffers in the TAsyncSerTool seem to be clear now. I reverse engineered the EmptyInputBuffer method more thoroughly and found out that the comm tool tries to get as much data into the NewtonScript buffers. There is still something to do about data which won’t fit… it looks like the async tool will lose some of that data because it might be overwritten. Or maybe it will use flow control. Anyway, I’ll just store it in an intermediate buffer which is hopefully big enough for all higher level protocols. For OBEX, it is 2k, and PPP should be happy with 1.5k. Having this now out of the way I can continue working on the Bluetooth setup application.

Now where did that byte go?

Thu, 30 Oct 2003 00:00:00 +0000

I’m currently working on the Bluetooth setup application, or to be more precise, on the device discovery feature. The Bluetooth setup application will be used to discover devices, pair with them and set system-wide parameters such as connection speeds.

In the process, I hit one of the tougher problems in projects like this: Figuring out what the existing NewtonOS code expects to happen in a new component (such as a new comm tool). It’s a very tedious process… currently, there seems to be an off-by-one error in the data transfer back to the NewtonScript world. For certain input scripts (mostly those which read byte by byte), the last byte in a transmission is lost. This bug also causes the PPP negotiation phase to take forever.

I also still haven’t found out why PPP is so slow. It seems that the peer device simply takes its time talking to the Newton. At least on the PPP and HCI level, I can’t see any retransmits or packet errors. But I’m working on it.

Rough Edges

Sun, 26 Oct 2003 00:00:00 +0000

With Courier 0.5 out, I’ve got now again some more patience for working on Blunt. Caching and download were something that just had to be in Courier, but now it’s Hammer time again.

One remarkable step forward today was a successful PPP connection over Bluetooth to my Linux machine. There is however something seriously wrong with either the MTU size, the ppp daemon or flow control: Transfer speed is about 400 bytes per second. Not really breathtaking. That prompted me to get OBEX working reliably to get some alternative numbers. There were (and probably are) still some rough edges in the L2CAP and RFCOMM layer. A remaining mystery is why some packets get garbled right after connection establishment. But simply sending them again helps.

Anyway, the speed over OBEX is more satisfactory: about 10-15k/sec from my Mac, which is quite good considering that the serial speed to the Bluetooth card is 23k/sec. I guess that the slow PPP speed is related to the failure of establishing a PPP connection to my 3650…

LPR Driver

Thu, 23 Oct 2003 00:00:00 +0000

There was an interesting discussion on NewtonTalk on printing. I do not print very much anymore - it was exciting in the beginning, with devices such as the toilet roll printer that worked with the ZX Spectrum, or the first inkjet printers - but the amount of material I would have to print for doing my work is simply too high.

But on the other hand, a challenge is a challenge, no matter how you look at it! So I investigated how much it would take to enable TCP/IP (or rather, lpr) printing from the Newton. And it seems quite feasible. Praise the NewtonOS again for that. A lpr printer driver would be a subclass of the TPSPrinterDriver protocol class, similar to the TPSPAPPrinter which does AppleTalk printing. I played around a bit, and got a dummy printer class working. It only needs the code to initiate the connection and send some data. Unfortunately, connecting over TCP/IP requires grabbing a link and doing a DNS lookup, something which wasn’t really planned in the current printing process. I guess it would have to be done via a proto patch of the printing transport…

Caching

Tue, 21 Oct 2003 00:00:00 +0000

I’ve been fiddeling around with Courier over the last couple of days. One of the planned improvements is caching of pages in the history and it seems to work reasonably well now. One problem however resurfaced: The protoTXView can use a VBO to store large text files, but it seems to be somewhat unstable. Sometimes, the stored data gets corrupted, resulting in -10600 and -10061 errors. The workaround for now is to have either history caching or VBO usage.

'Let things settle'

Sun, 12 Oct 2003 00:00:00 +0000

Sometimes, things just don’t want to work as designed, you encounter Heisenbugs, random errors show up, and finding a rational explanation is way too much work. In these cases, “let things settle” is a perfectly valid strategy! L2CAP connection establishment was a bit flakey in Blunt, so I added a command to get the link quality before initiating the connection, delaying things a bit. Seems to work so far. And I got the first bytes sent over from the Newton as a connecting device to the Mac machine.

Starting a PPP daemon on the Mac to let the Newton establish a TCP/IP connection via Bluetooth should be possible now… but there is still lots of additional code required, especially the Internet Setup/Blunt coordination is missing (stuff like peer discovery, finding the correct RFCOMM port, handling pairing etc.)

Pieces of the Puzzle

Sat, 11 Oct 2003 00:00:00 +0000

Connecting to a Linux machine via Bluetooth works now. Data transfer seems to work as well, but that’s not really verified. The 3650 and Mac OS X don’t like some of the connection parameters used in the top level RFCOMM layer, but much of the lower layers is implemented. Some mysteries still need to be solved, for example the timing when sending packets via the HCI layer. Sometimes, the packets get lost… what would help is using delay timers, but that’s an area I’ve tried to avoid so far. Timer events should however be a universal feature on the NewtonOS, and they would help in making the Bluetooth connection more reliable.

I also cracked open a no-name USB Bluetooth dongle to see what kind of module is inside. Unfortunately, Bluetooth modules often come in either UART or USB flavor, and the module I found is a USB only module. So much for getting cheap internal Bluetooth on the Newton! The modules from BTDesigner.com are not that expensive, but shipping is about $30…

One thing that got a bit in the way of my development work is that Mac OS X 10.2.8 broke my Bluetooth setup. The system preferences crash in the Bluetooth panel (yes, I’ve deleted all relevant .plist files) and my Acer dongle doesn’t work anymore. Seems like Apple wants people to upgrade to Panther.

Buffer Sizes and intriguing thoughts

Thu, 02 Oct 2003 00:00:00 +0000

I’m almost ready to release a proof-of-concept version of Blunt. It still crashes occasionally and sometimes, a connection can’t be established from Mac OS X. But otherwise, I’ve been able to successfully send files from my 3650 and desktop to the Newton. The bug du jour was a buffer overflow, causing some bytes to be lost here and there, but that’s fixed now. The buffer sizes in the various places in the stack definitely need some adjusting!

An interesting related topic are of course Bluetooth cards. The PICO card which I’m using is neither very battery-friendly nor small, so a nice, small, low-power alternative would be great. And it doesn’t have to be a PCMCIA or CF card: It should be possible to put together a small PCB with a Bluetooth module and fit it inside the Newton, similar to the SER-001. Bluetooth modules are tiny, all-in-one solutions which can be interfaced with using serial lines and which don’t need very much extra circuitry…

Wow!

Tue, 23 Sep 2003 00:00:00 +0000

Wrestling with buffer management between NewtonScript and Blunt … getting hit by flow control problems … wondering where all the bytes went … crashing and crashing and crashing …

… and then a quick, stupid test: Why not let Neo instantiate an RFCOMM endpoint instead of an IrDA endpoint? Change one line, add some instantiation options, initiate receiving on the Newton, open Bluetooth File Exchange on the Mac, hit send … it works!

Now the first file to be ever transferred over Bluetooth from a Mac to a Newton sits in the Newton’s inbox. Quite neat. I guess that’s worth a nice Westmalle Dubbel.

Seven Bytes

Sun, 21 Sep 2003 00:00:00 +0000

… made it safely from the Mac OS X Bluetooth File Exchange application through various layers of protocol handling code in Blunt to show up in a text view on the Newton. The seven bytes are an OBEX connection request.

Now I’ve got to add code for sending data back, check the necessary flow control mechanism and the first third of the Bluetooth integration is done. It would already allow sending files to the Newton via Bluetooth. The second third is initiating connections instead of listening and the last third is cleaning up things (mostly SDP) and adding a setup application.

Shudder!

Fri, 12 Sep 2003 00:00:00 +0000

Who did this!? It’s like talking in gzip! … what I’m talking about is the Service Discovery Protocol (SDP), used to find the services running on top of a Bluetooth stack. It uses a very compact binary data representation which might be space efficient but very inconvenient to parse. Oh well.

On the brighter side of things, I’ve managed to restructure some of the higher level Blunt code without breaking too much. Now there is some degree of multiplexing possible. I also had the idea of using a permanently operating Bluetooth stack which could have multiple comm tools as clients. This could be really interesting to run multiple services on the Newton as well as sending data at the same time. I have to remember why that idea didn’t work with IrDA, there must have been a catch.

Totally unrelated, but in case anybody missed it (that’s not really possible, is it - there is this really practical RSS channel on the 40Hz SourceForgepage !): Courier 0.3 and Raissa 1.1 are out.

'Pairing was successful.'

Wed, 10 Sep 2003 00:00:00 +0000

From a Newton to Mac OS X and a Nokia 3650, using Blunt. Not too bad. At the moment, the code looks a bit hacked together (well, that’s what it is), and the usual “hack - refactor - hack” cycle has to start sometime. But for now, it’s good enough. Next up is initiating connections.

Steady Progress

Thu, 04 Sep 2003 00:00:00 +0000

At the moment, it’s fill-in-the-blanks for Blunt. The Bluetooth specification is quite clear, I have two peer devices to test with, the comm tool integration is working so far - now all those functions, states and packet formats have to be implemented! I can already accept incoming connections at the HCI level, but there is much more to come, such as the service discovery protocol.

Blunt architecture

Mon, 01 Sep 2003 00:00:00 +0000

Bluetooth support will consist of two main pieces: The RFCOMM communications layer (realized as a comm tool) and the Bluetooth Setup application. The RFCOMM layer will handle HCI and L2CAP to provide a serial port over Bluetooth. The Bluetooth Setup application will handle the rest, including discovery and pairing. It can actually be implemented in NewtonScript, I’ll have to see about that.

There is also the area of Bluetooth by-products. At the moment, I’m thinking mostly about location-based services. It would be quite simple to let the Newton scan its Bluetooth neighborhood upon wakeup to determine where it is at the moment. It is also easy to advertise its presence by simply being discoverable for some time. All this could be done at the NewtonScript level.

Unfortunately, the current architecture restricts usage of the serial port to one program at a time. As an example, a location-based service would have to share the port with the receiving transport and the sending transport (Neo). Unless I figure out something smart such as multiplexing at the Endpoint level…

Blunt

Sun, 10 Aug 2003 00:00:00 +0000

Blunt - the Bluetooth driver for the Newton - is taking shape. The first fragments are now on the Experiments page. Today’s problem was how to get the HCI layer called… the original TAsyncSerTool handles input only if there is an active InputSpec on the NewtonScript side. For now, the approach is to fake that we are in the receiving state. Seems to work, at least I get periodic results from a Bluetooth discovery process.

Communication Protocols 101

Fri, 08 Aug 2003 00:00:00 +0000

One of the challenges in the Bluetooth driver project and any other project involving the NewtonOS communications framework are the communication primitives which are exposed to the NewtonScript layer.

These primitives (Bind, Connect, Input, Output etc) have to be implemented by any lower layer protocol such as RCOMM or SSL. It is of course possible to bypass them and offer a separate API via a native module (I can already talk to my Bluetooth card on that level), but then, no existing application could take advantage of the new protocol.

The RFCOMM layer has to fit under the generic TCommTool interface which in turn is called using the outer communications primitives. In addition to that, there is a tradeoff in reusing the existing code for interacting with the serial chip (implementing in TSerTool and TAsyncSerTool) and being able to implement whatever buffering scheme and packet handler is necessary for the HCI/L2CAP layer.

I figure that reusing the TAsyncSerTool is the best way to go. It implements a significant amount of logic and I don’t think I could reimplement a comm tool from scratch. This means that I have to reverse engineer it to that extent that I know how to fit the HCI/L2CAP layers into the existing code.

From the experiences with Nitro I have already figured out which functions are called in which order when a NewtonScript client uses an endpoint. The situation regarding the RFCOMM tool is quite similar: The TSerTool handles the upper layer input buffers which are visible to the NewtonScript layer. The TAsyncSerTool uses its own ring buffers to talk to the hardware and fill the TSerTool buffers. This is point where the RFCOMM tool will come into play. Instead of simply copying the data between the buffers, it will have to implement the HCI/L2CAP logic. This requires changes to the IRQ handler, the DoInput and DoOutput functions and a number of internal functions (which are thankfully all declared as virtual so they can be overloaded).

The biggest difficulty is to implement the logic which the NewtonScript glue layer expects from the comm tool layer…

I thought I'd share this

Tue, 05 Aug 2003 00:00:00 +0000

Look at the Experiments page. Courier and ntox 2.0 are still not finished and tested enough, but surely, somebody will find them at least interesting ;)

The plan

Thu, 31 Jul 2003 00:00:00 +0000

It seems that many of my assumptions about a Bluetooth driver for the Newton were correct and it is indeed possible to implement such a thing. The basic idea is to implement a new comm tool based on the TAsyncSerTool which would provide the RFCOMM layer. This is very similar to the work I’ve done on Nitro, it might even be a bit cleaner … and run on on older MessagePads as well.

One common low-level interface to Bluetooth cards is a UART. This means that we can simply talk to the card via a serial channel. There are different kinds of UARTs - if a card would use a UART the built-in TSerialChip driver won’t recognize, we would have to implement a new TSerialChip driver. But that is fortunately not required, the Newton seems to recognize most UARTs just fine.

The next layer is the Bluetooth HCI layer. It is a set of commands and events encapsulated in data packets. It provides functions to establish connections, do the key and PIN code negotiations, transmit data and so forth.

On top of the HCI layer goes the L2CAP layer. It provides logical connections over a physcal HCI connection. This will be the basis for the RFCOMM layer.

'A communications card has been inserted' ...

Wed, 30 Jul 2003 00:00:00 +0000

… let the fun begin! The card is a PICO PCMCIA Type II Bluetooth card. Currently, there are large numbers being sold on eBay in Germany, so I invested the donations I’ve received so far (thanks guys!) and got one for myself.

This means that I can start with the probably insane task of implementing a Bluetooth stack for the Newton. At least the RFCOMM part…

As for the cards that finally might be supported: They have to be PCMCIA Type II cards (16 bit, 5 Volt - 3.3 Volt might work too). CardBus cards such as the newer Belkin models won’t work.

Some weird stuff

Mon, 21 Jul 2003 00:00:00 +0000

I’m not sure if it is useful for anybody, but I just uploaded my old serial test code and the modified Endpoint code. It’s under Various Experiments.

The Neo/Nitro split

Sun, 22 Jun 2003 00:00:00 +0000

Since syncing turned out to be less satisfactory than I thought, I started to add the missing TCP/IP transport to Nitro. Taking the OBEX layer out of Nitro was a logical consequence, with Neo as the result. The TinyTP/IrCOMM part of Nitro has been stable for quite a while now, warranting a 1.0 version number. Taking OBEX out of Nitro also make bug tracking easier.

Neo contains the previously released IrOBEX transport and a new TCPOBEX transport. The TCP transport is far from perfect (no status dialogs, no receiving), but it allows you to send vCard, iCalendar and other data to a desktop computer. Too bad that the available OBEX solutions for desktop computers are a bit unfinished… But OpenOBEX is good enough for that task.

Later, Neo might also contain an RFOBEX transport, if and when there is Bluetooth support on the Newton. We’ll see…

IrDA, IrOBEX, Nitro, IC/VC and the Newton

Sat, 24 May 2003 00:00:00 +0000

Nitro and IC/VC try to follow the spirit of the original Newton architecture by reusing the existing interfaces you already know (hopefully): The Transports, the Inbox, the Action button and the Routing Slips. Here’s an overview:

A transport is responsible to send and receive data. Transports are for example Mail, Beam or IrOBEX. You chose a transport via the Action button (the one with the Envelope)
The Inbox is used by transports to store incoming data. You can tell which transport put something in the Inbox by looking at the item’s icon. If there is an application on the Newton that knows what to do with the item, it will add a “Put Away” choice to the menu behind the Inbox button with the tag on it.
The Action button is used to do something to the currently selected item in an application. There are three groups of functions available: Things like Duplicate and Delete, sending via a transport and application defined actions. Sending via a transport works by selecting the item you want to send, choosing the transport from the Action button menu and specifying the sending options in the Routing Slip.
The Routing slip is used by transports to get more information necessary for sending an item. This might include the address where to send to (for Mail) or the data format in which the item shall be sent.

Now how does all this apply to IrDA et al.? Quite simple:

The transport which Nitro provides is called IrOBEX. IrOBEX is a standard understood by most IrDA capable devices. It is quite similar to HTTP.
If you want to receive items via IrOBEX, you’d use the Inbox. There, IrOBEX is offered as a choice in the Receive menu. If you’re receiving things from a Windows PC, you have to modify the IrOBEX preferences (accessible in the Inbox [i ] menu) to use OBEX:IrXfer. This is because Microsoft sucks and doesn’t like to conform to standards.
The Action button is what you use to send items via IrOBEX. Just select the item to be sent and chose IrOBEX from the Action button menu.
The Routing slip comes into play if you have installed IC/VC. IC/VC offers the iCalendar/vCalendar and vCard data formats. You should use these if you want to import the data into the peer devices’s applications, e.g. an address book or calendar application.
IC/VC also adds the capability to put away vCalendar, iCalendar, vCard and Newton packages from the Inbox after they have been received via IrOBEX. Please check the IC/VC preferences in the Inbox (under the [i ] menu) to specify what should happen if there are conflicts between received items and items already on the Newton.

Some examples:

You want to send a Names entry to your T68i. You open Names and select the entry to be sent. Then you tap the Action button and select IrOBEX. Because you want to use the entry in the address book on the phone, you select vCard as the data format in the Routing slip. Then you tap on Send, select Now and point your Newton to the T68i which has IrDA reception enabled.
You want to receive a calendar entry from your phone. First, you open the Inbox. Then you select IrOBEX from the Receive menu. Point your phone to the Newton and initiate sending from the phone. After the item has been received, tap on the Tag button and chose Put Away. In the following dialog, chose IC/VC as the target.
You want to export all your Names entries to your desktop PC via email. You select all Names entries in Names, tap on the Action button and chose Mail as the transport. You fill out the necessary fields in the following Routing slip, select vCard as the data format and tap on Send. The email will be one vCard file which you can save as one file on the desktop PC and import it into programs that know how to deal with vCard files. Microsoft programs don’t fall into this category because they can’t handle more than one vCard in one file (there are however workarounds).

In no particular order

Wed, 30 Apr 2003 00:00:00 +0000

I think I have to write this down so I don’t forget it:

Improve Patch for NIE Modem and Serial Support
Generic Package installer for Inbox (part of IrOBEX?): check.
New UI for MAD Max
Fix MAD Max truncated playback bug (schedule shorter sounds?): check.
SMS sending transport to be used with Nitro
Delay automatic disconnect on no IAS activity in Nitro
MasterCook MX2 file reader: check.
C++-based HTTP library for improved streaming
Reverse engineer full C++ codec interface to schedule sounds from C++
Reverse engineer AIFtoNTK to use ARM tools
Use ARM tools to compile MAD Newton
Port faac (AAC decoder) and Tremor (Ogg Vorbis decoder)
Fast HTML renderer plugin for NetHopper: check (Courier)
Speaker dependent speech recognition using EARS
Bluetooth CommTool driver for RFCOMM
Time and position dependent application scripter
FAT/FAT32 file reading in MAD Max via ATA support
Buy lots of chocolate and/or beer

Something completely different

Tue, 29 Apr 2003 00:00:00 +0000

… well, not really. But I took a closer look at the latest ARM compilers to find out how to make them work with the Newton tools. The advantage of using the ARM compilers are probably better code generation and definitely the support for StrongARM specific opcodes. The most interesting case are the 64 bit multiplication operations and support for 64 bit integers which are more or less required for optimized MP3 and Ogg decoding (you see where I’m getting at…)

Turns out that it is not impossible to use the new compilers, but also not trivial. The first hurdle is of course how to get the compilers. Thankfully, I can play with the latest tools at work, including the ADS tools. But the next step is a bit more challenging. The ARM tools are a distinct relative of the Norcroft tools used by the NCT. They differ however in the file formats they use. The ARM tools are nowadays ELF-based, whereas the Norcroft tools used AIF and AOF as object formats. This requires that the complete compilation and linking process has to be done with the same tool chain. Switching file formats in between stages doesn’t really work that well…

The last step in that process is the conversion from an AIF image to the format used by the NTK. This step is done by the AIFtoNTK tool that unfortunately does not like the AIF image created by the new ARMLink and FromELF tools. Add the different debugging symbols to the mix and you end up with a total mess. But all is not lost, the AIFtoNTK tool might be easy to reverse engineer (it is only a couple of dozen kilobytes large). It takes the code, relocation and debugging symbol areas from the AIF and puts them in an NTK wrapper. That shouldn’t be too hard to reimplement…

However, there are still quite a few other projects I’d like to improve before I get back to this.

Using Nitro

Thu, 17 Apr 2003 00:00:00 +0000

Since I’ve got the GPRS settings for my Nokia 3650 correct, I was able to use Nitro a bit more extensively. This included using Raissa, SimpleMail, Nethopper and NewtScape. Overall, it went quite smoothly, no crashes and more or less acceptable speed (the 115kbps figure for IrDA is just theoretical, the protocol overhead is quite big).

It seems that at least for me, the combination of a Newton and a cell phone covers most use cases. It is actually strange that the situation has gotten better recently with the rise of RSS. It allows me to fetch information in a compact way, without having to visit huge, feature-laden web sites… but I still have to put together a package with some sample RSS channels.

[if you looking for a tutorial about using Nitro and IC/VC, here it is. ]

Why it works the way it works

Wed, 09 Apr 2003 00:00:00 +0000

With the release of IC/VC (the iCalendar/vCard route layer), some explanation of the used mechanisms might be helpful. The button with the little envelope which you see in your Newton applications is actually a very powerful tool. It is a system wide context menu that allows to perform actions based on the type of data currently displayed. The actions are either application defined or available throughout the system. The latter ones are clearly more interesting.

Most Newton applications follow the same principle. They display a list of items which are stored in a soup and can be modified individually. Every item is contained in a frame and typed via the ‘class slot. The system wide actions are available based on this type. The most common use for these system wide actions is to send the current item via a transport to somewhere else. Other actions are possible as well (e.g. reading an item via MacInTalk), but they are also based on the item’s type.

I’ll cover the transport case here. A transport advertises which data types it can handle. The most common data types are ’text and ‘frame, but there can also be others. Indeed, if you look at the Names or Dates soup, you’ll notice that there are ‘person, ‘company and ‘meeting types. Does the transport have to know about all of these? Not at all. It would be quite difficult to implement for example an email transport that would have to translate every possible data type on the Newton into an email message.

Instead, route scripts are used. They provide the translation between the application specific data types and the more general “transportable” data types. IC/VC is an example of such a route script. It registers as a handler for the data types used by Names and Dates and provides them in a form Nitro can handle them. The OBEX part of Nitro declares that it will handle the ’text datatype, so that’s what IC/VC provides. With a little twist: types on the Newton can have subtypes, and IC/VC does not provide an object of the type ’text, but instead ’text/calendar or ’text/directory. The neat thing about that is that it is still text and can be send via any transport that handles text. Nitro takes the subtype into account and sets the appropriate OBEX headers according to that.

So what happens now when a user taps the action button? First, the system adds all the application defined actions to the popup menu. Then, it identifies the type of the current object (I’ll spare you the details what happens in an app’s overview). Based on that type, it constructs a list of route scripts that can handle the data type. The final step is to then determine which data type each route script will provide and compile a list of transports that will handle these data types. These transports will be added to the popup menu.

The double indirection of route scripts and transports is really well done and is one of the things that puts the Newton OS still so much ahead of the rest of the PDA world. And that five years after is discontinuation…

Timing is crucial

Mon, 07 Apr 2003 00:00:00 +0000

Windows is again driving me crazy. First is the stupid default usage of OBEX:IrXfer instead of OBEX, including undocumented error codes, and now it’s the timing when sending items to the Newton. Well, the Newton IrDA stack has some flaws too, but only with Windows it’s so tricky. The issue is receiving items which works now a bit differently than the standard procedure (at least on Windows 2000). The problem is that the Newton listens for an incoming connection, accepts it, waits for some action and disconnects if nothing happens.

Turns out that the way that Windows was designed to work clashes with this. If you initiate reception on the Newton, the IrDA activity icon is shown on the Windows task bar. Only then the option to send an item will be added to the context menu in the Explorer. By the time you have waited for that and activated it, the Newton as already disconnected, thinking the peer fell asleep.

The workaround is to open the wireless file transfer panel on the Windows side before that, navigating to the file to be transferred, initiating the reception on the Newton and then sending the item from Windows… not nice, but it works - besides the fact that there is probably still a protocol error in Nitro that I have to fix.

That squishing sound...

Wed, 02 Apr 2003 00:00:00 +0000

… you are hearing are bugs being eliminated. One by one, but going forward steadily. I just uploaded Nitro 0.2.2 which should be more robust and compatible. The most important change in the Nitro core is that no more out-of-band TinyTP and IrCOMM packets are sent. This might complicate matters in those cases where the peer runs out of packets or wants to use the simulated modem lines, but it prevents Nitro from crashing peer devices due to malformed packets. I might have to look into that later and re-enable these packets, but for now, I can reliably use my 3650 as a cell phone modem.

The second bigger change was the addition of the OBEX:IrXfer option for sending and receiving data. It is needed because the Newton can only advertise one IrDA class when listening, and in some cases, there is a mismatch between the class the peer device expects and the class the Newton advertises (e.g. when sending from Windows 98). The correct behaviour on the Newton side would be to allow more classes but there seems to be a hardcoded logic that only the class that has actually been set via the NewtonScript endpoint interface will be accepted. Other classes and attributes can be added to the IAS server, but they are ignored for the LSAP selector (which determines the class).

Some bugs have been fixed regarding handling of OBEX data, hopefully increasing compatibility…

Handling Unicode

Tue, 01 Apr 2003 00:00:00 +0000

One very nice thing about the Newton OS is that it is fully Unicode-based. Unfortunately, this raises some issues when communicating with other devices which are not Unicode-aware… in addition to that, Unicode comes in different flavors which all want to be dealt with.

Handling Unicode in IrOBEX transactions requires actions when sending and receiving data that has a textual representation. This includes plain text, HTML, iCalendar and vCard data. My initial strategy for the sending part is that all text based routing data (e.g. all the textual representations generated by the standard route scripts) will be sent as Unicode, to be more specific, UTF-16 Big Endian. This is the native format on the Newton. Other routing formats such as vCard and iCalendar will most likely be UTF-8. Hopefully the receiving end of this will handle that accordingly…

Receiving data is probably a bit easier, I have already implemented a function to recognize and convert ISO8859, UTF-16LE and UTF-16BE for text data. Memory consumption is however a concern, that is not optimized yet.

Nitro 0.2.1

Tue, 25 Mar 2003 00:00:00 +0000

Just finished the latest version of Nitro. Quite some work has gone into the OBEX layer, but it is indeed a nice test of the lower layers of Nitro as well. There are still bugs in that lower layer that can cause nasty crashes or stalled communication, but I hope to get those ironed out eventually. One interesting discovery I made while debugging with Hammer is that the IAS implementation on the Newton is actually quite complete. IAS is used to advertise the IrDA services and parameters to peer devices, one example are the device name but also IrCOMM parameters. The part that made people believe the IrDA stack on the Newton is incomplete is only the NewtonScript interface. The rule to remember here: You don’t know anything about a system unless you have really gone down to the metal!

The OBEX implementation in Nitro can be used also via other transports. It might be interesting to use TCP/IP or a serial connection. At least for Unix, there is Open OBEX that could be used on the peer side.

Clearly the next step after fixing the bugs is to implement the missing route scripts. Beaming text back and forth gets boring after some time! Sending iCalendar and vCard data is probably very easy, and some initial tests have shown that the routing stuff on the Newton is very well thought out. As soon as a route script implements the vCard or iCalendar conversion, it can be used over any transport, making it e.g. possible to send iCalendar entries via email.

Tedious Transport

Wed, 19 Mar 2003 00:00:00 +0000

The whole high level communications infrastructure on the Newton is fairly complex. It takes place mostly in the NewtonScript layer and is therefore built on cooperative multitasking: Instead of using threads and semaphores, callbacks are used. This requires much more careful design…

Anyway, the OBEX transport now works over IrDA. It does not handle error cases gracefully and contains no user feedback. But it is integrated in the Newton’s I/O box mechanism so sending and receiving over IrOBEX just looks like any other transport (e.g. email). I can currently send text files to my cell phone or a desktop PC. This includes everything that has a textual representation on the Newton: Notes, Names, RSS feeds, … Receiving files is not implemented yet but is hopefully not too difficult.

It will get interesting as soon as more routing formats are supported. iCalendar and vCard are one example. Routing formats are independent from the transport and can be used over any transport. The sending part of iCalendar and vCard is next on my todo list.

BEGIN:VCALENDAR...

Tue, 18 Mar 2003 00:00:00 +0000

… that’s the output I just got in my Inspector window. Communication via IrOBEX between my Newton and a Nokia 3650 works. Implemented at the moment are an OBEX server with the Put method and an OBEX client with Get and Put methods. OBEX is transport neutral, but as I wrote before, it is a nice test case for Nitro. It actually confirmed that I can set up a listening TinyTP endpoint (works for OBEX but probably not for IrCOMM which is of no big use in that case anyway) which I haven’t tried before.

This is the plan: Wrap the OBEX client and server into a Transport for sending and receiving any file type via IrDA and TCP/IP. Implement a Routing layer for iCalendar and vCard datatypes (encoding and decoding), including a merge heuristic for incoming iCalendar and vCard data. Result: Making my Newton more usable in daily life, effectively saving its butt - I’m still resisting to buy a cheap Zaurus ;)

Controlled Crashes...

Tue, 11 Mar 2003 00:00:00 +0000

… are something nice. There was an interesting problem with Nitro when connecting to a Windows 2000 machine, causing an immediate crash. This was easy to reproduce and catch with Hammer. The problem was an IAP query frame sent to the Newton after successful connection. It seems that the lower levels of the Newton’s IrDA stack can’t handle this. The IAP frame was sent because the IrDA stack on Windows 2000 constantly sends out active connection requests which of course contain these frames. If the Newton also initiates a connection, then there is a chance that these frames get received at the wrong time in the protocol.

The solution is to defuse the stray IAP frame and turn it into a innocent IAP acknowledgement frame. I’ve still got to test this further but it doesn’t seem to cause any damage. The result is uploaded as version 0.1.2. It would be nice if people who encountered crashes upon connection could try if it helps.

Version 0.1.2 also contains code to connect to other peer classes such as OBEX or IrLPT. This is based on the connection options passed to the NewtonScript endpoint. Nitro Test demonstrates this feature.

Some Updates

Mon, 10 Mar 2003 00:00:00 +0000

I’ve been busy working on OBEX (currently over TCP/IP, but it is transport independent). OBEX should be a valid test case for the TinyTP layer of Nitro. Nitro seems to be working best on Siemens phones and worst on Ericsson models. Testing this is pretty complicated since the means to signal an error condition (or even logging some data) from within a comm tool are quite limited. For now, the focus will be on catching potentially critical states in the protocol and not do something stupid.

Interestingly enough, Nitro seems to work my Nokia 3650 at least to some extent. It connects via a regular GSM data connection, but then the phone crashes after transferring some data. I’ll have to try that with a more recent firmware for the Nokia.

I’ve also been able to finally try a USB/IrDA adapter under Mac OS X 10.1.5, and unfortunately, it doesn’t work at all. I suspect that this is due to the adapter (an Actisys IR4000US), which isn’t even supported under Linux. Other adapters might work - Apple is even selling them via their European web store.

Bugs and Crashes

Wed, 26 Feb 2003 00:00:00 +0000

Now that the IrCOMM code will be used more widely, there should be an explanation about the current bugs and why crashes occur.

The most common symptoms of something going wrong are:

a spontaneous restart of the Newton
an error message saying “Sorry a problem has occurred” with a positive, random error number
stalled communication between the Newton and the peer device
crash of the peer device
no connection to the peer device
failure of an upper level protocol (PPP, TCP/IP, HTTP)

Symptoms 1 and 2 hint at a mistreatment of some of the internal IrDA data. One example is the treatment of a IrLMP control packet as a data packet. I’ve tried to make the code robust for these cases but it depends mostly on the packet types sent by the peer device (and thus on the peer’s IrDA stack implementation) and the level of my successful reverse engineering of the IrDA and comm layer.

Symptom 3 happens when there is either a protocol error (i.e. one side sends packets the other side doesn’t handle in its current state) or, more likely, there are not enough resources on the Newton to deal with the packet. A typical situation is receiving lots of small packets and handling each packet in a time consuming way. The layers of the comm system are not completely decoupled, meaning that an upper layer can effectively block the whole stack. If that happens, the Newton sends RNR packets to the peer, telling that it is currently not able to handle data packets. The cure for a situation like this is to increase the number and size of the IrDA receive buffers (currently, I use seven 512 byte buffers).

Symptom 4 is causes by sending a malformed IrCOMM or TinyTP packet. Although I tried to catch all packets leaving the Newton to transform them to correct IrCOMM packets, there might still be a place somewhere where a packet is passed through an unknown channel (this happened with the PPP driver talking directly to the serial comm tool).

Symptom 5 is usually a sign that the peer device and the Newton disagree with IrDA service to use. Currently, the Newton requests the IrDA:IrCOMM service over TinyTP. Peer devices can however provide IrLPT or something else and as a consequence, deny the connection.

Symptom 6 is unrelated to IrCOMM, but it could still be caused by errors in the Nitro code, e.g. when packet data is corruptet.

Anyway, I’m working on all of these…

0.1

Mon, 24 Feb 2003 00:00:00 +0000

This is it: Nitro. It comes with source code so you can take a look at all the stuff I recently wrote about. I will also put the hacked TSerialEndpoint and the TSerialChip implementation on the web at some time.

Nitro for now advertises it’s services under the “ircm” label. Any application that wants to make an IrCOMM connection should use this as the service identifier. For applications that haven’t been modified (e.g. the NIE Modem and Serial support module), patching the application package by replacing the Unicode string “aser” or “irda” with “ircm” should do the trick (Unicode strings use two-byte characters, in this case the high byte is zero and the low byte is the ASCII code). I’ve provided a patched NIE Modem & Serial module on Nitro’s web page. I’ll try LookOut as soon as I receive my USB IrDA dongle.

Eric Schneck (of iTunes Plug-in fame) has had success using Nitro with his Motorola Timeport T280, Earthlink and SimpleMail. I have currently some trouble for establishing a PPP connection with a Nokia 3650 with Radiolinja as the provider, so this calls for some logging capabilities to find out what exactly happens.

Other missing pieces are a working OBEX layer (which shouldn’t be to difficult), the associated data handlers and a way to not only initiate but also accept incoming connections.

Success!

Sun, 23 Feb 2003 00:00:00 +0000

Task almost complete: I just got a working PPP connection via IrDA from my trusty development MP2100 to a Linux box. Including fetching a web page, so there was also some data going back and forth. There is still some serious testing left (including other devices such as cell phones), and some features like hardware flow control are missing for now, but this is definitely a nice success!

Anatomy of the TIrDATool

Thu, 20 Feb 2003 00:00:00 +0000

The latest difficulty (as if there haven’t been enough) was to find a place where to handle the IrCOMM specific data in the regular IrDA data flow. My first idea was to use the StartOutput and GetComplete funcitons as hooks. But again, the NewtonScript layer caused some trouble. To get it working anyway, the inner workings of the TIrDATool had to be exposed a bit more.

The TIrDATool class is derived from TAsyncSerTool. TAsyncSerTool provides asynchronous serial I/O via two ring buffers (TCircleBuf class), one for reading, one for writing. On the MP2x000, the ring buffers are filled by the Voyager chipset using DMA.

The two scenarios that had to be investigated further are reading and writing. In the TIrDATool class, this involves copying data from the ring buffers to the buffers used by the comm layer (and thus by the NewtonScript layer). These buffers are parameters to the StartInput and StartOutput functions. They are instances of the CBufferList class and are directly linked to the parameters given to the Input and Output functions of the associated NewtonScript endpoint. And this is also the reason why hooking into StartOutput and GetComplete doesn’t work quite well: These buffers do not reflect any of the packet-related characteristics of the IrDA data flow, they are merely holding the final streams of data.

So there has to be a way to get into the data flow from the ring buffers to the target buffers. This data transfer is done in the DoOutput and DoInput methods of TIrDATool. In fact, these methods are called for any IR data transfer, including the low level IrLMP and IrLAP packets. Quite handy.

Since there is some protocol overhead and logic for IrDA, the TIrDATool uses a number of helper classes. The most important one is TIrSIR. TIrSIR uses TIrLAPPutBuffer for any outgoing requests and contains a CBufferSegment for the incoming data. The task is now to put the IrCOMM data into the TIrLAPPutBuffer before sending and removing the IrCOMM data from the CBufferSegment after receiving.

Quick Update

Thu, 13 Feb 2003 00:00:00 +0000

Implementing the IrCOMM comm tool seems to go quite well. The first step was to get all the base classes of TIrDATool correctly declared. Lots of digging through C++ vtable declarations… I’ve got this love/have relationship with C++, at the moment I’m happy how easy it is to reverse engineer and patch it. And how cross-platform and cross-compiler it is.

Much of the added logic goes into the ConnectionComplete, StartOutput and GetComplete functions of the new comm tool. Sending the IrCOMM negotiation packet right after connecting works already, receiving the answer is however still a bit unclear. I hope that the comm tool mechanism lets me schedule something via StartInput. Otherwise, a timer event based mechanism like the one I tried with the serial endpoint event handler might be an alternative.

The NewtonOS Communications System

Wed, 12 Feb 2003 00:00:00 +0000

I guess a bit of clarification regarding the whole comms system on the Newton might be useful at this point. The big picture is still unclear at some points and might even be incorrect here and there, but this is how it is supposed to work…

A communications mechanism is encapsulated in a comm tool. There are several tools on the Newton, including the serial and IrDA tool. They are derived from the class TCommTool. For the IrDA tool, the full inheritance chain is: TUTaskWorld, TCommTool, TSerTool, TAsyncSerTool and finally TIrDATool. Comm tool classes make use of virtual functions to overload needed functionality (good for hacking too). A comm tool is running in its own task.

Comm tools are instantiated via an associated service. For the IrDA tool, this is the TIrDAService class. A service is an implementation of the TCMService protocol. When declaring a service class, the capabilities macro has to be used to state which service this class provides. This is also the parameter used when instantiating an endpoint.

Services are managed by the Communications Manager. The CM is not implemented in a single class but instead via several global functions and classes. The CMGetEndpoint function is the starting point for all endpoint communication. It triggers starting the necessary comm tool task by looking up the requested service and calling the Start method.

Endpoints and comm tools are bound together via an event handler (TEndpointEventHandler). It is important to know that an endpoint can be used by the NewtonScript task and the comm tool task, requiring synchronisation to prevent crashes.

On the lowest level, a comm tool uses hardware specific drivers to perform the actual I/O operations. The drivers can be layered, in the IrDA cases it is the TIrGlue class talking to a serial chip that controls the Ir LED. The TSerialChip protocol is an example for such a driver.

So far, I have been able to implement, replace or modify some of the more interesting classes in this picture: TSerialChip, TSerialEndpoint, TEndpointEventHandler and finally, TIrDATool. This would eventually pave the way for stuff like Bluetooth, SSL or SSH.

More Asynchronous Stuff

Thu, 06 Feb 2003 00:00:00 +0000

As it turns out, the NIE Modem & Serial support is a good test case. It uses only asynchronous function calls, including nBind and nConnect. Binding an endpoint is not an issue, but as you might remember, an IrCOMM connect requires some handshaking. Also, the connection options are a bit different.

Two problems need to be solved to make an asynchronous connect work: Find a place where to do the IrCOMM handshake and manage the IrCOMM connection options in a way that it does not break anything else. The option management is indeed quite interesting in that the class that controls a TSerialEndpoint from the NewtonScript side evaluates the result of the connect and passes it on to the NewtonScript layer. For that purpose, it uses the options objects set in the nConnect call. To make the IrCOMM stuff play nice with this mechanism, the IrCOMM connect options have to be removed before controll passes on…

The asynchronous IrCOMM handshake will be handled either before the first send or receive operation. Another possibility is staring it from the endpoint’s event handling routine.

IrCOMM working in NewtonScript

Tue, 04 Feb 2003 00:00:00 +0000

Good news: A previously dumb IrDA endpoint now knows all about TinyTP and IrCOMM. This is on the NewtonScript side of things, so if you now instantiate an IrDA endpoint, you can initiate an IrCOMM connection (notice that accepting IrCOMM connections is not done yet).

There are some things to note regarding the current implementation: If the code is active, all IrDA connections are IrCOMM connections. This will could get in the way of existing applications such as beaming. It is a bit difficult to anticipate when to use IrCOMM and when to use IrLMP (the Newton’s default). For that reason, I will add a preference setting to turn on IrCOMM manually.

It is also possible to talk to another IrCOMM peer directly using the C++ interface. This is easier because it uses synchronous communication. I’ll add a NewtonScript interface to these functions as well.

One big piece of the puzzle is however currently missing: how to make existing applications that use a plain serial connection (i.e. not IrDA) to use IrCOMM? I was hacking the NIE Modem/Serial support package a bit, changing the instantiation parameters from ‘aser’ (serial connection) to ‘irda’ (IrDA connection) and the physical location of the port from ’extr’ (external port) to ‘infr’ (IR led). It didn’t really do the job, but some parts of the IrCOMM code came already to life. Time to set some breakpoints and find out what’s going on!

Near Misses

Sun, 02 Feb 2003 00:00:00 +0000

A pretty nasty problem showed up that required going deeper into the endpoint mechanism: Asynchronous data transfer. It is needed because the NewtonScript world works based on the execution of small, non-blocking pieces of code that either return quickly or kick off some sort of background task (such as the whole endpoint infrastructure). One of the central points of data handling in a NewtonScript-based communication application is therefore the InputScript that processes incoming data received by an endpoint. There is no blocking read. This is something to get used to and confuses programmers regularily. For that reason, more “recent” example code from Apple introduced finite state machines which deal with these asynchronous events on the NewtonScript side. In the C++ world, most things are however pretty standard (one reason why a rewrite of parts of NHttpLib, Raissa and MAD Max in C++ could be very promising).

The problem in the IrCOMM case is that data reception is not handled by the nRcv function in the endpoint but in the endpoint’s event handler code. I hadn’t looked into that mechanism before, but fortunately, the event handling code is part of the TEndpoint protocol interface and can be overridden. In the process of mapping the mechanism, some of the classes and data structures had to be reverse engineered as well. It was not too painful - usually, it is possible to gain some understanding by looking at the C++ constructor and the initialization code. But nevertheless, I think that reverse engineering data structures is more tedious than trying to understand code.

Asynchronous reception of data works now. With one little problem left: sending a packet when handling an incoming packet doesn’t seem to work correctly. It is needed to advance TinyTP credits to the peer. The situation becomes critical when a large amount of data is sent from the peer. If the peer doesn’t get any credit, it just stops. The solution to this is to either find a way to send data properly in the event handler or set up some sort of idle handler in a different thread that checks the remote credit from time to time and sends a packet if needed. Time to look at more event handler code!

Connect works!

Fri, 31 Jan 2003 00:00:00 +0000

The first step in adding the necessary IrCOMM code to the modified TSerialEndpoint is done. The modified endpoint will be instantiated whenever a regular serial connection (IrDA, modem and serial) is made. In addition to that, connecting to another IrCOMM device over IrDA works. Next are send and receive and then we’re done. Connecting is probably the more critical operation anyway since it contains already sending and receiving of data.

In the first incarnation, IrCOMM will work when using a regular IrDA connection. This means that an application that wants to use IrCOMM has to request an IrDA connection. For legacy applications, there are two options: Either patch the application to use IrDA instead of serial (it could be as easy as changing four bytes in the package), or go the hard way and patch the TSerialEndpoint further so that it uses a second hidden IrDA endpoint when requesting regular serial connections…

Crazy Idea No. 1

Wed, 29 Jan 2003 00:00:00 +0000

The existing IrDA tool on OS 2.1 creates a TSerialEndpoint when an application requests an IrDA connection. As we know, there is a little more massaging necessary so that the data that is sent over this endpoint conforms to TinyTP and IrCOMM. The same goes for receiving. In addition to that, data has to be sent in packets.

After the failed attempt to make the IrCOMM implementation look like a TSerialChip, the next crazy idea is to make it look like a TSerialEndpoint. This is several layers higher in the Newton’s comm stack and contains code that is called from the NewtonScript task (making it hopefully a bit less sensitive). The pitfall here is that the TinyTP layer must be able to receive a packet while sending and vice versa. Speaking in Endpoint terms, a Snd/nSnd must be possible right after a Rcv/nRcv, and a Rcv/nRcv might be needed before a Snd/nSnd without returning to the NewtonScript task. A workaround for this might be asynchronous send and receive operations.

An endpoint on the Newton is an implementation of the TEndpoint protocol class. Creating such an implementation involves providing a class declaration and definition which will get processed by the ProtocolGen tool, resulting in assembler glue code. The packager tool then prepares the implementation so that it will be registered automatically when the containing package is loaded onto the Newton. A registered protocol implementation can be instantiated using NewByName. Protocols can also be unregistered. This provides a nice way to extend or configure the system.

The problem in the current situation is however that the existing TSerialEndpoint code is to be reused since I do not intend to reimplement the rest of its functionality. But subclassing from protocol implementation classes is not possible so we have to chose a different approach here. First, we simply define the class TSerialEndpoint as it is currently implemented in the OS. When this declaration is passed to ProtocolGen, the resulting glue code references the existing TSerialEndpoint code (of course this requires setting up a jump table similar to CardGlue.a). Second, we override the necessary send and receive functions with our code. And third, we deregister the existing TSerialEndpoint protocol and register our own.

The first and third step already work and the new code gets called. The challenge is now to weave the IrCOMM implementation into this…

Deeper and Deeper

Tue, 28 Jan 2003 00:00:00 +0000

To make things a bit more interesting, there are two different versions of the TSerialChip class (to be more precise, it is actually a protocol - I might explain the term protocol later, but it is also described in the DDK). The newer one (version 2.0) seems to override the older one completely, but it is also more complex. My original plan was to simply use the older one but that caused just some fancy crashes here and there.

Using the new verion of the TSerialChip protocol is unfortunately not so straightforward for various reasons. The most important one is that any implementation of the V2.0 protocol has to be provided with a so called capability. It tells that it does indeed support the new protocol via a static data field added by the ProtocolGen tool at compile time. The data to be set is declared in a macro in the C++ class declaration, but to figure out the actual syntax of that macro, a fun little exercise of reverse engineering the ProtocolGen tool on the Macintosh is needed. HexEdit is your friend in these situations and fortunately, the Apple engineers have used a string represenation of the grammar for ProtocolGen. So the capabilities macro goes into the class declaration without a trailing semicolon. Good to know.

The current hacked dummy TSerialChip implementation now already gets pretty far. On the way, we lost our friend the TLog class (a simple utility class by Paul Guyot that uses the NameServer for logging), probably due to some concurrency problems. But Hammer works nicely and it looks like the usual Instantate/Bind/Connect sequence works for this dummy implementation. A bit later, it crashes but that is not too scary. As opposed to the stuff I saw that was going on between the different layers of the comms system. Let’s hope that this is reentrant!

NIE and IrCOMM

Fri, 24 Jan 2003 00:00:00 +0000

Looks like we’re one step closer to getting NIE to connect via IrDA. A little testing showed that my previous assumptions on how to disguise the IrCOMM implementation as a serial chip are true. The only major showstopper might be that the Newton’s communication system does not like it to have its ears stiched to… you know what! Then, the only way out would be to port a complete IrDA stack and sign into the next lunatic asylum. But for Bluetooth, this might be a whole different story and well worth it.

Newton, meet GPRS

Wed, 22 Jan 2003 00:00:00 +0000

Funny how small and seemingly stupid things can get you excited. But my cell phone has just responded to an “AT” with a nice and clean “OK”. So I got IrCOMM working between my Nokia cell phone and the Newton. Who would have thought that after all the negative statements about the Newton’s IrDA stack.

Of course, I also had to try what happens when I send an “ATDT*99#” over to the phone. And indeed it answers with the typical PPP line garbage. Leaving me with the task of disguising the IrCOMM endpoint as a serial chip so it can be used as a modem. More testing of the flow control and buffer management is also needed. For that, playing around with IrOBEX might be nice.

Code Cleanup

Tue, 21 Jan 2003 00:00:00 +0000

Nothing special today… it is time to clean up the prototype code I made for testing TinyTP and IrCOMM. This is never nice since you go from something working to something most likely broken. Restructuring always causes this first phase of putting stupid errors in the code, fixing them and then wondering how one can be so stupid to put them in in the first place.

But it always pays off. I think that developers who are afraid of throwing out prototype code and start from scratch with a new design are more likely to run into problems later on. However, when restarting from scratch, there is one thing I really wish for: A versioning system that is smart enough to take a snapshot of my current development setup, letting me peek back to find out what I did to make things work. Maybe a small script would help that archives a whole directory tree. The difficult part in this is unfortunately to figure out what to archive. And how to archive across multiple machines…

Well, the TinyTP stuff seems to work now and I’m satisfied with the low level interface. The only open issue is to figure out the initial credit the Newton receives from the peer. It is sent in the IrLMP connection response frame, for now I just use the standard value of 14. Different stacks might have different values here, but the Symbian and Linux stacks use 14.

IrDA on 2.1 Devices (Part 3)

Mon, 20 Jan 2003 00:00:00 +0000

The most interesting parts of the IrDA stack are IrCOMM and IrOBEX. IrCOMM emulates a serial connection and IrOBEX facilitates exchange of chunks of data, similar to HTTP. IrCOMM and IrOBEX are usually implemented on top of TinyTP (there is one variant of IrCOMM that uses IrLMP directly).

The goal is to enable the Newton to use IrCOMM instead of one of the serial ports (built-in port, PCMCIA serial cards, PCMCIA modem cards) to connect to other IrCOMM enabled devices. Most interestingly, this would be cell phones which can be used as wireless modems. I don’t have any firm plans for IrOBEX, but we’ll see…

IrCOMM follows the same principle as most other communications protocols: Use an existing layer, build on top of that and put any specific control data into the data part of the underlying frame format. IrCOMM introduces a number of control structures which are mostly used for flow control. It can emulate basic serial three wire without flow control, three wire serial with XON/XOFF flow control, nine wire serial with hardware flow control, or Centronics flow control. The control parameters are put into the beginning of a TinyTP frame, just after the TinyTP credit byte. The structure is simple: first a byte that tells how big the control structure is, and then (parameter type, parameter length, parameter data)-tuples. All this is defined in the IrCOMM specification.

To make IrCOMM work on the Newton, we need to have a working TinyTP layer. As noted before, IrCOMM will most likely work only when initiating the connection. But that should be ok for our purposes. The IrCOMM layer will open a regular TinyTP session to the peer’s IrDA:IrCOMM service. The IrCOMM service we request will be nine wire serial with hardware flow control, but we’ll indicate that we won’t use flow control. Maybe at some point later it can be added, but for now, I’ll try to go without it.

The remainder of the work is to make IrCOMM available as a regular serial port. For that, we have to go under the hood of the Newton Communications Manager. Fun stuff ahead!

IrDA on 2.1 Devices (Part 2)

Fri, 17 Jan 2003 00:00:00 +0000

The most important missing parts of the Newton’s IrDA stack are TinyTP, IrCOMM and IrOBEX. Today, I’ll explain briefly TinyTP. TinyTP is the simplest protocol of the missing ones and is required for IrCOMM and IrOBEX. It ensures flow control by using so called credits that are exchanged between the peers. One single credit entitles a party to send one more IrLMP frame to the peer. If you run out of credits, you cannot send any packets any more and have to wait for a packet giving you fresh credits. The credits which are advanced to the peer are stored in the first byte of an IrLMP frame.

Implementing TinyTP on the Newton is not too difficult. It requires two things: One, the packet size has to be known to locate the credit byte (it’s always at the beginning of the packet), and two, we have to be able to send the initial credit when opening a connection. The first task is achieved by getting the packet size via the kCMOIrDAReceiveBuffers option. The second task is being done by using the kCMOIrDAConnectUserData option to pass in a single byte in the connect frame. It sets the initial credits for the peer and is typically set to 14.

A read via a TinyTP endpoint has to always request the full packet size. If there is not enough data, the endpoint will simply return a smaller packet. If we want to read more data, we have to do that in chunks the size of a packet. So some buffer management is needed. Of course also the TinyTP credit management, which comes into play when we want to send something or the peer has run out of credits while sending. But basically, as soon as we have the packet size and can send initial connection data, TinyTP can be implemented. I have put together a small test application that verifies this and it does look promising. Of course all the implementation work will be in C++ since we want to get down to the actual endpoint interface!

Next will be an explanation of IrCOMM…

IrDA on 2.1 Devices (Part 1)

Thu, 16 Jan 2003 00:00:00 +0000

There seems to be a certail level of confusion about the actual IrDA capabilities of the MP2000, MP2100 and eMate, i.e. devices running NewtonOS 2.1. To make things short: OS 2.1 implements an almost complete IrDA stack up until IrLMP.

Now the long version of the story: IrDA support can actually mean a lot of things. In practice, an IrDA stack usually consists of at least three layers: IrPHY, IrLAP and IrLMP. Since it is a stack, these are layered on top of each other, meaning that for example IrLMP sends and receives data via IrLAP but adds its own control data to it. Other interesting layers are TinyTP, IrCOMM (both used for simple data transfer) and IrOBEX (exchange of structured objects).

IrPHY is the lowest layer and handles access to the send/receive hardware. In the case of the Newton, it is most likely a serial chip of some sorts connected to the IR diode. Meaning that you send data one byte at a time. On top of IrPHY sits IrLAP, the Link Access Protocol. It introduces the notion of a frame which contains three parts: An address field to specify who is the recipient, a control field that defines the purpose of the frame and a payload area that contains the data to be transmitted.

Now to IrLMP, the Link Management Protocol. It is of special interest to us since this is what we can use on the Newton. IrLMP is probably the most complex layer in the whole stack. Among other things, it introduces the Link Multiplex Sevice (multiple logical connections over one physical connection) and the Information Access Service (what services are offered by a specific device, e.g. IrCOMM or IrOBEX). The IAS can be seen as a simple directory that can be queried to find out more about the capabilities of the peer. Is organized in key/value pairs (so called Parameters) which are associated with a specific service (aka Class).

But what do we now do with all this? Simple: The IrDA stack on the Newton can be used to connect to any other IrDA-compatible device. No exceptions. Sounds nice, doesn’t it? But there is a catch: Other devices may have difficulties initiating a connection to the Newton and requesting a service other than TinyTP and IrOBEX. The reason is that the Newton’s IrDA stack can associate only one parameter with each service it provides. And some services require more than one parameter. Also, the service that the other device may connect to must be known beforehand. But overall, this is not a total catastrophy I would say, since actively connecting from the Newton side should still work (it definitely does for TinyTP, IrCOMM and IrOBEX).

More to follow…

What is this about?

Wed, 15 Jan 2003 00:00:00 +0000

Before the real fun starts, maybe some introductory words are in order… this web log is intended to document my successful and unsuccessful attempts at hacking the Apple Newton MessagePad. The Newton is indeed a strange beast, its latest incarnation (the MP2100) so far ahead of the time when it was axed that it is still fascinating to develop for it.

But as you will see from this log, fascination often turns into frustration, mostly caused by the lack of proper development documentation. And then it is time to fire up a hex editor, a hex calculator, the Hammer debugger, dig into ancient issues of the Newton Tech Journal, scan UNNA for example code, look at Usenet posts that will soon be over a decade old and hope that at the end of the day, there is a cold beer in the fridge to stop the madness…

PS: In case you wonder, the word mottek comes from a certain German dialect which was spoken by who you could most closely describe as vagabounds. It means hammer.