Alas, it wasn’t the famous box that opened – that one is still sitting sadly on a Parisian bookshelf, waiting patiently for its newlywed owners to transport it to their romantic island. No, the box that was solved is the black floral-patterned one from a brand new collection of prototypes I’ve recently constructed. These new models have features that improve on the original without damaging its charm or mystique.

The "Black Floral" prototype box

The new designs sport a discreet USB connector that provides the emergency “back door” access – for those that know the secret password. As the owner, you can open and close the box to your heart’s content and program your own destinations. When your recipient finishes his/her puzzle box adventure, you lock up a new “treasure” and begin another!

I was privileged to be in attendance at the big event, and when that little internal motor whirred to announce the opening, the car erupted with cheers. The experience taught me something interesting. I learned that besides hobbyists, geocachers, romantics, puzzle collectors, and artsy geeks, another population thinks puzzle boxing is pretty fun – kids!

The claimants to the title are in fact children — our two and one of their school friends. Here’s what happened.

I sealed a $20 bill in the “Black Floral” and programmed it to open at P. Terry’s, a slightly chic burger joint not too far from our house – 1.71 miles, in fact, according to the blue display. I modified the greeting to read “Your/dinner/is in/the box!” – customizing messages is another thing the owner can do – and handed it without a word of explanation to my 11-year-old son.

P. Terry's

Well, my kids know what to do with a puzzle box, and soon a van full of youngsters was motoring about town hunting for their supper, yours truly acting as silent chauffeur. As we drove, I discovered that young folks take a vastly different approach to solving this kind of puzzle. Most of my adult readers like drawing circles on maps and performing calculations vaguely suggestive of what we call trilateration, but kids seem to have little patience for pencils and compasses. They prefer to just drive, drive, drive – repeatedly pressing the button and changing course when the distances start increasing. This semi-random-walk technique can quickly get you stuck on a dead-end street in the “wrong” part of town, but I have to say it’s no less exciting than the scientific approach.

After a few faulty turns and considerable head scratching, they finally found their dinner. At one point in the adventure we actually stopped at a light immediately opposite P. Terry’s. I thought the gig was surely up, but even though the display read “Distance 163 feet” – just barely outside the 150 foot radius I had set – the kids still managed to fritter away 20 more hungry minutes navigating a parking lot half a mile further down William Cannon Drive.

Now that I’ve experienced the dual joys of building and delivering Reverse Geocache Puzzles, it’s very nice to actually get to play with them. Today my little nephew Nate was looking a bit forlorn, so I put a tiny toy in one of the small maple boxes and sent him on his own private mission. To boost his spirits I personalized his name into the greeting – “Hi Nate!/ Can you/open the/box..?”. He’s not yet eight, so I just sent him to a neighbor’s house about 600 feet away. When after a few minutes he came trotting back, Piglet style, “his” box open and treasure in hand, the smile on his face belied his happiness.

The Reverse Geocache continues to prove an interesting little puzzle with broad appeal. Whether it’s a romantic engagement-ring trek to the summit of the Matterhorn or a little kid’s 20-minute “adventure” down the street, solving one is a rich and joyful experience.

if (need_to_check_GPS)
{
  NewSoftSerial gpsconn(6, 3);
  ...
} // Object is destroyed/cleaned up here

Another useful new feature is the support for serial devices which use inverted TTL signalling — where “HIGH” is 0 and “LOW” 1. To enable automatic inversion, simply provide a boolean “true” as the third parameter in the constructor:

NewSoftSerial nssinv(3, 2, true); //uses inverted signalling

Could someone please test? I don’t have any inverse logic devices.

Many thanks to Garrett Mace and Paul Stoffregen for their contributions!

Update: Please upgrade to version 10c.  10b was a defective posting.

Mikal

The puzzle box continues to generate lots of interest.  This week I encountered it in two different geocaching podcasts, a Serbian blog, and Daniel Soltis’ talk at Playful ‘09 in London.  I’ve fielded calls from network TV channels, film producers, engineers, and machine shops all over the country.  Everyone seems to want to bottle up the magic that swirls around this strange little box.

I particularly enjoyed reading Soltis’ presentation, which focused on “the delight of physical objects connected to games.”  Tangible things like the puzzle box, he writes, provide “specificity, tactile pleasure, and possibilities for kinds of game play that multi-purpose platforms like phones and computers can’t.”  How true.  People love the connection to the physical, especially if there is a touch of mystery to it.  Would you even be reading this if I had just written some old GPS iPhone app?

From Nick Marsh's Notes for Playful '09

“Yes, yes, but have they managed to open the box yet?!”

As a matter of fact, I have recently been in very close email contact with Chris - that’s the newly-married friend who has unknowingly been at the center of this hullaballoo.  He writes to say that he has been busy putting finishing touches on his beautiful new film The Luminiferous Aether, preparing it for submission to festivals in the coming year.  I hope you get to see it someday– it’s a remarkable work, especially for a first effort.  This is the movie, incidentally, that led to romance with his leading lady those many months ago on the French island.  I was treated to a sneak preview shortly after I got back from their wedding, but I’m afraid the detailed appraisal I promised in return is dreadfully late.  There’s a whiff of irony here.  While Chris waits patiently for the thoughtful and sober review his film deserves, I’ve been too busy returning calls and composing articles about his puzzle box to write it.  He has no idea how much he himself, in some warped sense, is to blame for my slow response.

So to answer the question, no, I’m afraid I don’t have a single tidbit of new puzzle box gossip to share.  I haven’t heard a thing.  But isn’t it fun being part of the watching crowd?  I love it.  This surprising little adventure seems to be entering a third phase: first there was the creation and the presentation, then the unexpected storm of publicity; now we’re to the point where thousands of people - perhaps tens of thousands - are watching to see how it all plays out.  If you are reading this, you are now part of the story.  We are all initiates into what I am calling the “voyeuristic world”, reminiscent of the film The Truman Show, in which a young man inadvertently discovers that his daily life is being watched by millions on TV. The difference is that this reality show is real!

The Couple at the Wedding Dinner (Kelly Neal)

I had a nice conversation this week with photographer Kelly Neal, who graciously gave me free rein to use pictures like the above to illustrate my articles.  If the puzzle box story is moving too slowly to satisfy your voyeuristic needs, perhaps you’d enjoy strolling through his official wedding site, imagining what it must have been like to have attended that little French village wedding.  You are, after all, a part of the story!  If you like the pictures, please give Kelly a call.

And now some answers:

“Why don’t you just ask about the box?”

Well, because I’m enjoying the adventure too much.  I fear that asking my friends for an update will ultimately help hasten the story’s denouement.  I intend to ride it as far as it will go.

“Why do you think they haven’t said anything?

I think they’ve probably figured out where to go by now.  But it is 400 kilometers away, and they are busy people.  A trip like that would require at least a long, unencumbered weekend.  I imagine they are just waiting for a convenient time.

“What if the couple finds this web page…? Won’t that spoil it?

I think that’s part of the fascination.  Will they get to the island first… or find the web article?  I don’t know.  But even if they do discover the publicity, I’m not sure that will entirely spoil it.  They still need to get the box open, and there are still one or two things my article doesn’t reveal… the contents of the “overly sentimental card” for example.  I think there’s still plenty of mystery and suspense to go round.

“What happens if they don’t figure it out in 50 tries?”

Ah!  It displays a painful little message indicating that the box is sealed forever, then suggests returning it to the manufacturer for help.

“How can I get one?”

The puzzle box is not just a thing, it’s an experience.  It’s personal.  It’s mysterious.  It’s an adventure.  I don’t want to see glossy plastic versions mass produced all over the world, because such things would only dimly reflect the magic of the box I built for my friends and subtract from its unique beauty.  The original was lovingly constructed by a human being and embodies the special relationship between three people.  I don’t think such a personal thing should be turned into a commodity. 

That said, I have a team of people considering how we might build a small number for special, personal occasions.  Many people have had the idea, for example, of using a ring box to propose marriage.  Bravo!  Or how about this?  A proud father gives a box to a beloved daughter leaving home for college.  The box eventually leads her to a car dealership, revealing a set of keys.  The possibilities for a rich experience seem boundless.  I hope eventually to help people realize experiences like these.  Just don’t expect to see them anytime soon.  Or on the shelves at Target.

“Can I commission one?”

I am making a very small number of hand-built boxes, either replicas of the original elephant box or custom designs.  If you are serious, please contact me at mikal (at) arduiniana.org.

Where can I see The Luminiferous Aether?

Stay tuned.  It’s not available yet.

How do you pronounce your name?

The same way ordinary Michaels do.  Mine is simply a creative spelling of the common name.  Thanks, Mom and Dad!

If you want to keep following this story, subscribe to the RSS feed (above). I’m also trying out twitter @mikalhart. Leave a note anytime.

[Next read the most recent chapter in the puzzle box story.]

The Puzzle Box story has a delightful second act.

It’s Saturday night, five weeks to the day since the wedding, and despite the recent firestorm of publicity, I still haven’t heard whether my friends have managed to open the Reverse Geocache Puzzle Box.  It’s certainly possible they are playing it cool, waiting for just the right moment to share the story of their adventure, but I think it’s more likely the box is still intact.  If so, each passing day increases the likelihood that they will stumble first onto one of the dozens of articles that popped up on the web this week.

On Monday, Hack-a-Day and MAKE Magazine independently posted generous write ups on their blogs.  Then around noon Engadget, Slashgear, and Gizmodo also picked up the story of the mysterious box that can only be opened in one place on earth.  This seemed to trigger a flurry of tweets, links, and mentions on Facebook, various geocaching and wedding forums, blogs, and podcasts everywhere.  Sites as remote as China started running variants of the story - I recognized it only by the links and photos - and by Monday night my website had seen three times as much traffic in a single day than it had the entire year!

But a bigger push was still to come.  Late Wednesday I got a call to do a telephone interview with Los Angeles Times tech reporter David Colker, who weaved together a charming article comparing the box saga to “Mission Impossible” or James Bond.  David said that everyone in his office “just loved” it, both the tech-y folks and the not-so.  But that’s the appeal of this little project, isn’t it?  It works hard to balance the geeky with the intensely personal.  Or as my favorite tweeter wrote, “it’s the perfect combination of mystery, software, romance, and mechatronics.”  Ha!  I love that!  Yes friends, after centuries of seeking the ideal balance between those four great pillars of art, only now has perfection finally been achieved!

Thursday morning the Times ran the article, which was subsequently lifted by the Seattle Times, the Orlando Sentinel, the New York Post, and a number of West-coast blogs. Traffic on arduiniana.org surged even higher.  And despite all, we still don’t know whether the couple have opened the box.  It’s like the world has been partitioned.  Those who have seen the amazing coverage stand on the one side, eagerly waiting to see when and how those on the other side - my friend and his small circle - will stumble across the secret.  Personally, I hope that he opens the box before discovering all the press.

And so we keep waiting.

Meanwhile, I have enjoyed reading all the supportive comments.  It’s humbling to see adjectives like “dazzling”, “delightful”, and “amazing” bandied about in reference to one’s work.  People seemed especially to like the story of the man who called me a BASTARD! and my friend exhorting everyone “Don’t push the button!”  Digging through the photos posted on the wedding photographer’s website, I was delighted to see that he captured the very instant that my friend first uttered that phrase, “Ne poussez pas le bouton!”  The anxious speaker, unfortunately, is off camera right in this photo, but you can see my cousin Scott (right) and myself (standing) about to burst with laughter.

Hey, dont push that button!

And here, for your edification, is the BASTARD guy himself, facing the camera:

The guy who called me a BASTARD!

And still we wait.

Be assured that as soon as I hear how this story ends, I will let you all know.  One reader suggested that the couple should put the box up for sale on EBay.  Lots of people want copies of the Reverse Geocache Puzzle box, but owning the famous original - now that should be worth something!

Heady days.

Mikal

[Next read chapter 3 in the puzzle box story.]

The Puzzle Box

There is a cool new addition to the website today — a project I call the Reverse Geocache Puzzle (tm).  It’s a box that won’t open until its owner figures out where on earth to move it.  I built it for a friend who got married in France last month.  The software for this project incorporates all five of the major libraries on Arduiniana.  Let me know what you think.

Mikal

What does that give you?  Well, a couple of things.  Most importantly, it makes NewSoftSerial more compatible with the latest Arduino release.  Changes to timings of digitalRead and digitalWrite in Arduino 0015 broke NewSoftSerial at higher baud rates.  All software serial libraries are timing sensitive, and even a microsecond or two change in a function’s timing can introduce enough error to fatally corrupt serial transmissions.  By using direct port I/O and retuning all the timing tables, we avoid problems caused by the longer digitalReads and digitalWrites.  Furthermore, the library should be more immune to future changes of this kind.

Direct port I/O is amazingly fast — about 10-25 times faster than the equivalent call to digitalRead/Write.  Because of this, NewSoftSerial 9 is able to operate reliably at higher baud rates than in the past.  In my test suite, I am now able to get reliable RX at speeds up to an unprecedented 57.6K baud on a 16MHz processor and 31.25K baud at 8MHz*.  Because the performance paths no longer use digitalRead/Write, they are less sluggish and much easier to tune to the higher speeds.

*These figures are about 70% higher than NewSoftSerial 8 with Arduino 0014.  These were measured using USB-powered Arduinos running a prerelease v. 0016 copy of the software, which optimizes the timer 0 overflow counter — an important contributor to software serial performance.  I/O was tightly controlled to avoid overflow, which can easily occur at high baud rates when you have no hardware flow control.

Please let me know what you think.  The new version is here.

What’s next for NewSoftSerial?  Well, there are a number of things being asked for:

• Arduino Mega support
• Signal inversion option
• RTS/CTS flow control option
• Configurability, i.e. “E, 7, 1″ style parity, data and stopbit configuration.

What are you most interested in?  Let me know here.

Mikal

The good news is that version 9 has been optimized heavily and it should prove to be the best version ever — and run on both 0015 and 0014. Stay tuned!

Mikal

The half-speed clock makes accurate serial reception considerably more difficult, but NewSoftSerial 8 still manages to transmit with 100% reliability up to 57.6K baud and receive up to 14.4K baud on the Arduino Pro.  Reception at rates up to 38.4K baud is 99+% reliable.  For a better understanding of why the slower processor caps RX at a relatively low baud rate, and for a trick to significantly improve this, read on.

Flush

The new library supports the flush() method, which works the same way that HardwareSerial’s flush() does.

      void flush(); // discard the contents of the RX buffer.

PROGMEM use

NewSoftSerial 8 has been reorganized to move some of the logic into a PROGMEM-based lookup table. You may encounter the following spurious warning message when you build your first NewSoftSerial 8 application:

    NewSoftSerial.cpp:68: warning: only initialized variables
    can be placed into program memory area

Be assured that this message is completely harmless. As soon as I figure out how to remove it, I will.

Potential trouble spots

There are two problem areas that affect accuracy and cap RX speeds in all software serial libraries:

• interference from other interrupts in the system
• RX interrupt floods/bursts

Before I discuss these, an introduction to interrupt-driven serial reception may be in order.

Interrupt-driven RX

It all begins when the Arduino software RX pin, normally high, drops low. This is the “start bit” – the signal from the transmitter that bits of data are about to arrive in a sequence of pulses whose width is determined by the agreed-upon baud rate.  The pin’s state change triggers an interrupt which launches the NewSoftSerial recv() routine. Through a series of eight precisely timed samples, recv() assembles the new byte and stores it in a buffer before releasing the interrupt.

The top waveform in the image below shows the arrival of the letter ‘a’ (hex 61). The RX signal drops low (start bit) and then the data bits arrive, in order from lowest to highest, 1, 0, 0, 0, 0, 1, 1, 0, followed by a high “stop” bit. At each interval, the recv() routine samples the pin and records the bit value. The second waveform demonstrates this sampling process in action, one sample per falling edge.

Fig 1. The letter ‘a’ (hex 61) being sampled correctly

If all goes well, the received byte is identical to the one transmitted. And this is usually the case, because once the pin-change interrupt fires, NewSoftSerial’s precisely tuned recv() routine gains absolute control of the processor. Interrupts are disabled during the sampling, so there is no danger that a competing interrupt will corrupt the process.

The problem is that if another interrupt handler is already active when the start bit arrives, then NewSoftSerial’s pin-change handler is necessarily delayed until the competing routine exits.  If the delay is too long, the sampling routine may read the incoming byte incorrectly, causing it to be mangled or even lost.

When you encounter RX corruption or loss at high baud rates, the most common culprit is Arduino’s own timer overflow interrupt handler – the routine responsible for millis() and delay().  Consider: on an 8MHz Arduino the timer interrupt lasts a full 20µs, compared to only about 70µs for the entire width of a bit transmitted at 14.4K baud. If the timer interrupt fires at precisely the “wrong” time, i.e. just as a new start bit arrives, important processing may get delayed long enough to cause an error.

The picture below shows just such an event taking place. Comparing to the figure above, you’ll see that the system timer interrupt fires just before the “Serial” line goes low, shifting the sampling process to the right just enough to distort the sampled byte.

Fig 2. A timer interrupt fatally delaying the sampling sequence

In cases like this, you’ll experience an invalid byte read. On 16MHz processors, the experimentally determined failure rate is quite low – less than 0.5% – and is only seen at the very highest supported baud rates (57600 and 38400) .  But at 8MHz, the problem is exacerbated by the fact that the timer tick interrupt takes twice as long to execute.  Not only is there a greater likelihood that the interrupts will collide unhappily, but because the timer tick is longer, the damage done when there is a collision is proportionally greater.  What this means is that NewSoftSerial running at 8MHz receives data with 100% accuracy only at baud rates less than half those which work reliably at 16MHz. If you want perfect accuracy with an 8MHz clock, stick to rates of 19200 or 14400 baud or less.  Or…

Workaround: disabling the timer

…if your application can do without the system timer, it is possible to significantly improve RX reliability by turning it off.  NewSoftSerial 8 provides a new method to disable/enable the clock by masking off its interrupt:

      static void enable_timer0(bool enable);

While disabling the timer more than doubles the speeds at which 8MHz NewSoftSerial can reliably receive data, it is not recommended that you do so unless you understand the ramifications. Don’t do it unless you really need perfect reliability at speeds greater than 14400 baud and can live without millis() and delay().  Many applications can absorb the low failures rates (<1%) NewSoftSerial experiences at higher speeds.  For example, for serial GPS devices, TinyGPS automatically filters out NMEA sentences that fail the checksum.  This may be sufficient for your app… or not.

Flooding/Bursting

Another situation that can adversely affect data reception is when a peer sends rapid bursts of data (at any baud rate), flooding the Arduino. The bytes arrive one after another so quickly that NewSoftSerial stays constantly in the RX interrupt, filling the receive queue but never allowing it to be drained. This generates an overflow condition, which can be detected by examining NewSoftSerial’s overflow() routine – if your program can ever call it.

Even if it doesn’t trigger an overflow, a flood of bytes can cause problems at high speeds, because tiny errors in the timings that would otherwise have gone unnoticed accumulate.

The best way to avoid problems relating to “flooding” is to prevent them. Eventually I hope to add support for RTS/CTS flow control to NewSoftSerial. In the meanwhile, keep away from devices, especially high speed ones, that generate floods of data.

Timer based options?

I’ve had several discussions recently about whether it would be possible to implement a timer-based (as opposed to pin-change-based) software serial device in Arduino. Such a library would theoretically allow a number of simultaneously operating serial devices to coexist, because the interrupt times would be much shorter. However, based on my recent studies with the logic analyzer, I have begun to doubt whether a library that worked at reasonably high speeds could be built. There is just too much interference from other interrupts in the system.

Conclusions

Software serial devices are best running at lower baud rates, especially on 8MHz processors. If you need multiple reliable, high speed serial interfaces, consider ATmega328-based Duemilanove, which sports three hardware serial ports.

Future Directions

As I mentioned earlier, I will be working soon on implementing an RTS/CTS flow control mechanism to improve the interface with serial devices that tend to transmit in bursts.

The next version of NewSoftSerial will also likely support optional signal inversion.

A couple of people have also expressed interest in being able to more precisely configure the data and parity bits of serial protocol, especially, for example, the relatively popular “E/7/1”. It’s hard to assess exactly how desirable this is.  Who would support increasing the NewSoftSerial footprint by a few bytes in exchange for this kind of configurability?

Download

You can get the new version, as usual, in the “Libraries” section at NewSoftSerial.

Mikal Hart

With it, you can now write code like this:

Serial << "A is " << lettera << "." << endl;
lcd << "The date is " << d << "-" << m << "-" << y << ".";
  
eth << "You can use modifiers too, for example:" << endl;
Serial << _BYTE(a) << " is " << _HEX(a) << " in hex. " << endl;

And thanks very much for all the Arduinians who have used and commented on Streaming.  Cool stuff.

The new library is available in the “Libraries” section or here.

Mikal

for (int i=0; i<temperature_table.count(); ++i)
  if (temperature_table[i] > 98.6)
    Serial.println(temperature_table[i]);

Interested to learn more? Read all about it here!

Mikal