Flash

A Library to Ease Accessing Flash-based (PROGMEM) Data

Storing static program data in flash/PROGMEM is a tricky part of Arduino programming. To save precious RAM, a novice user already at odds with unfamiliar C++ syntax must digest such daunting concepts as prog_char, PSTR(), PROGMEM, pgm_read_word(), etc. Even seasoned users get tripped up by the indirection and typecasting that are required to retrieve valid PROGMEM data. Add to that a couple of apparent bugs in the implementation, and it’s clear that PROGMEM is a complicated mess.

I have written a new library, Flash, which abstracts away most of this complexity. It provides new String, Array, Table, and String Array types that make ROM-based data collections as easy to use as “normal” types. Each overrides the C++ [] operator, so to extract individual elements one uses familiar array access syntax:

// float array example
float t = temperatures[950];

// (2D) font table example
digitalWrite(pin[i], font_table['a'][i]);
// string example
for (int i=0; i<str.length(); ++i)
  if (str[i] == ';') break;

Creating named Flash objects

To create a Flash object, you use a library-provided macro. Each of the four basic types provides its own creation macro:

Strings: FLASH_STRING(name, value)
Arrays: FLASH_ARRAY(type, name, list of values…)
Tables: FLASH_TABLE(type, name, columns, values…)
String Arrays: FLASH_STRING_ARRAY(name, values…)

Examples:

FLASH_STRING(big_string, "Moreover, as you might appreciate, "
    "their implications were such as to provoke a certain "
    "degree of sorrow within me. Indeed - why should I not "
    "admit it? - at that moment, my heart was breaking.");
FLASH_ARRAY(float, temperatures, 23.1, 23.1, 23.2, 23.2, 23.4,
    23.7, 25.0, 26.0, 26.8, 28.8, 30.2, 31.9, 33.1, 33.1, 33.2,
    33.2, 33.4, 33.7, 35.0, 36.0, 36.8, 38.8, 40.2, 41.9);
FLASH_TABLE(boolean, font_table, 7, {0,0,0,0,0,0,0},
    {0,0,0,1,0,1,1}, {1,0,1,1,1,0,1}, {1,1,0,0,0,0,0},
    {0,1,0,1,0,1,0}, {1,0,1,1,1,0,1}, {1,0,0,1,0,0,1},
    {0,0,1,1,0,1,1}, {1,0,1,1,1,1,1});
FLASH_STRING_ARRAY(digits, PSTR("Zero"), PSTR("One"),
    PSTR("Two"), PSTR("Three"), PSTR("Four"), PSTR("Five"),
    PSTR("Six"), PSTR("Seven"), PSTR("Eight"), PSTR("Nine"));

[Note that you can make Arrays and Tables out of any native type.]

[Note also that the String Array is a slightly special case. For technical reasons, as you can see in the example, each member of the array initializer must be wrapped with the PSTR() operator. For this reason, String Arrays cannot be declared at global scope, but must be non-statically defined within in the body of a function. Note also that while the individual strings in a String Array are contained in PROGMEM space, the array itself (again, for technical reasons) is RAM-based, consuming 2 bytes per array element. This limitation does not apply to the other Flash types.]

Using named Flash objects

Array-like access. Each Flash type provides an [] operator to get access to the underlying data, for example:

    // extract an element from an Array
    float t = temperatures[5];
    // examine characters in a String
    for (int i=0; i<big_string.length(); ++i)
      if (big_string[i] == 'O') ++big_O_counter;
    // Use a font table to configure a 7-segment display
    for (int i=0; i<font_table[1].count(); ++i)
      digitalWrite(pins[i], font_table[1][i]);
    // Examine the sizes of strings in a String Arra
    for (int i=0; i<digits.count(); ++i)
      Serial.println(digits[i].length());

Since Tables are really just two-dimensional arrays, the Table object’s implementation of [] returns a one-dimensional Array object.

Size. Each Flash type provides a mechanism for determining its size:

String and String Array: size_t length();
Array: size_t count();
Table: size_t rows(); size_t cols();

Access. If you need it, each Flash object provides an access() method that returns the underlying PROGMEM pointer

Print compatibility. Each Flash object has a print() method that can be used to serialize the object to any stream derived from core class Print (HardwareSerial, NewSoftSerial, LiquidCrystal, Ethernet, PString, etc.) See “Printing Flash objects” below.

String copy. Flash String objects may be copied in whole or part to RAM using the String object’s copy method:

    // copies the first 10 characters of the big_string
    char temp[11];
    big_string.copy(temp, 10);

Printing Flash objects

Each Flash object can “print” itself, using either its print method directly:

    temperatures.print(Serial);
    big_string.print(lcd);
    font_table.print(newsoftserial);

or, equivalently, by using the << streaming operator:

    Serial << temperatures;
    lcd << big_string;
    newsoftserial << font_table;

Flash objects can also be “printed” using the familiar Serial.print[ln] syntax, but this requires a small patch to the core Print.h file. See “Patching Print.h” below

Using unnamed (inline) Flash Strings

One of the most useful features of the Flash library is the capability to print unnamed strings embedded directly within an expression. In other words, if you want to use a ROM string in a print statement, it is not necessary to declare a string variable and print it like this:

    FLASH_STRING(big_string12,
      "'Twas brillig and the slithy toves did gyre and gimble");
    lcd << big_string12;

Instead you can simply use the new built-in F() macro (F for “Flash”) to do the same thing inline:

    lcd << F("'Twas brillig and the slithy toves did gyre and gimble");

And again, you can also use

    lcd.print(F("'Twas brillig and the slithy toves did gyre and gimble"));

to print an inline string, but this requires the patch to Print.h.

Patching Print.h (optional)

By making a very small change to the core Print.h file, you can add the capability to print either named or unnamed Flash objects using the familiar print[ln] syntax. What the patch does is tell class Print about Flash objects and how to print them. Note that while this patch is useful, it is entirely optional. Flash already provides two mechanisms for printing ROM strings and data structures:

    obj.print(stream);
    stream << obj;

This patch adds a third technique, the familar

    stream.print(obj);

To patch, add the following interface definition to the top of Print.h, right after the #include statements. This defines what a Flash object is, from the perspective of the Print class.

    #define ARDUINO_CORE_PRINTABLE_SUPPORT
    class Print;
    class _Printable
    {
    public:
      virtual void print(Print &stream) const = 0;
    };

Then add a couple of inline print methods — alongside all the others — to the body of class Print:

    void println(const _Printable &obj)
    { obj.print(*this); println(); }
    void print(const _Printable &obj)
    { obj.print(*this); }

No need to recompile anything; you are now ready to roll:

    Serial.print(myarray);
    Serial.print(mytable);
    Serial.print(F("A long, long, unnamed inline ROM string"));

Library Version

You can retrieve the version of the Flash library by inspecting FLASH_LIBRARY_VERSION.

int ver = FLASH_LIBRARY_VERSION;

Download

The latest version of Flash is available here: Flash3.zip

Change Log

  1. initial version
  2. moved some code to a new .cpp file to avoid inlining every member function
  3. fixed a bug in the copy() method

Acknowledgements

Thanks to forum user Bill W. (”westfw”) for testing Flash and observing that Strings take up a surprising amount of extra PROGMEM space (20 bytes per string). I used his data to modify the library so that some of the commonly used functions, notably the _FLASH_STRING constructor are not inlined. This change reduces the footprint of the example programs noticeably. Thanks, Bill!

I appreciate any suggestions or input.

Mikal Hart

Page last updated on January 6, 2010 at 9:41 pm
20 Responses → “Flash”

  1. Coofer Cat

    3 months ago

    This library rocks! I’ve just moved all the long strings from my application to flash (saving around 100-200 bytes of RAM) in about 5 minutes! I wonder what else I can save…?

    I’m now looking at putting many more strings into my application, which I’d have had to live without before. One easy library just made my application many times more usable - thank you!


  2. Mikal

    3 months ago

    Hey– thanks for the nice feedback, Coofer!

    Mikal


  3. Christian

    3 months ago

    Mikal,
    This is a great library. How would you write back to one of the array elements? To clarify, let’s use your simple temparature array declration above. After I declare the array, if later in my sketch I wanted to change the value temperature[5], how would I do that? Could you please provide some sample code? I did not see any further documentation on the Access() function which it appears may be what I am looking for.

    Thanks in advance,
    Christian


  4. Mikal

    3 months ago

    Christian –

    Unfortunately, because of the nature of the architecture of flash memory, there is no way for a program to rewrite the flash values. They must be considered read-only by the software. If you want to store values in non-volatile memory, use the EEPROM library.

    Mikal


  5. Matt E

    2 months ago

    Awesome library!

    So far converting all my Serial.print(”"); commands to Serial << F(”"); has saved me 2/3 of my previous used RAM, though it seems the expected amount of saved RAM versus used Flash is not consistent. The first batch of commands I converted yielded an average 3.5:1 of used Flash to saved RAM. The second batch yielded an average of 7.9:1. I would have expected the second batch to have a better average as the flash library’s overhead would already be accounted for but that wasn’t the case.

    What would be the best method of comparing FLASH_STRING(string1,”Data here”); with char string2[]=”Data here”; in terms of RAM usage? I was using strcmp when comparing two whole strings and a nested for-loop when looking for a partial match.

    Cheers


  6. Mikal

    2 months ago

    Thank you, Matt.

    Yeah it would be nice if there were a built-in “compare” function to compare to strings. Without it, I guess the best thing is to write a for loop:

    for (int i=0; i<string1.size(); ++i)
    {
      if (string1[i] != string2[i])
        return false;
    }
    return true;

    Mikal


  7. topper

    2 months ago

    Thank you Mikal,
    I can now finish my Christmas lights easily thanks to your excellent work.

    Happy Christmas


  8. Mikal

    2 months ago

    Hey, thanks for sharing, topper! Happy Christmas to you as well!
    Best wishes,

    Mikal


  9. Johann

    1 month ago

    @Matt E: You can add the following to Flash.h:

    // compare to another string
      bool operator==(const char *str)
      { return compare(str)==0; }
    
      // compare to anoter string: less, equal, greater
      inline byte compare(const char *str)
      { return -strcmp_P(str, _arr); }

    it allows comparisons:
    (myFlashStr == “abc”)

    Thanks for a great library!

    -kellerza


  10. Mikal

    1 month ago

    Johann,

    Excellent contribution. I’ll add that into the next version. Thanks!

    Mikal


  11. Cort

    1 month ago

    Please excuse my lack of programming experience. I’ve been using this library with good luck so far, but have found one strange issue and a question.

    When I do two reads back to back, I have problems without doing something to delay in between them”

    ratemin = row_wave[0][0];
    delayMicroseconds(50);
    ratemax = row_wave[0][1];
    rate = random(ratemin, ratemax);

    If I take out the delayMicroseconds (just to give it something to do) the data reads don’t return the correct data. This is totally confusing to me.

    The question is, I’m trying to create pointers to FLASH_TABLE items I’d like to randomly select a table from a list, then pass it to a function. I can do this with a “normal” table without problem, but it doesn’t seem to work with FLASH_TABLE and I fear my skill level isn’t sufficient to determine if this is simply a limitation of the library, or I simply don’t know the syntax.

    Thanks, and again, Mikal, I very much appreciate the effort to hide PROGMEM from me!


  12. Mikal

    1 month ago

    Cort,

    I can’t understand why you might be experiencing differing behaviors when you add or remove the delay. Could you perhaps post the entire relevant code body? I don’t see this behavior.

    As far as the question about selecting a random table, suffice it to know that when you use the FLASH_TABLE macro, under the covers a template class of type _FLASH_TABLE<type> is created. So for example, here’s some code that randomly selects one of two tables and passes a reference to that table to a function, which then dumps some of it:

    #include <Flash.h>
    FLASH_TABLE(int, row_wave1, 4,
      {1,2,3,4}, {5,6,7,8}, {9,10,11,12}, {13,14,15,16});
    FLASH_TABLE(int, row_wave2, 4,
      {4,3,2,1}, {8,7,6,5}, {12,11,10,9}, {16,15,14,13});
    
    typedef _FLASH_TABLE<int> TABLETYPE;
    TABLETYPE *tables[] = {&row_wave1, &row_wave2};
    
    void myfunction(TABLETYPE &table)
    {
      Serial.print(table[0][0]);
      Serial.print(table[1][1]);
      Serial.print(table[2][2]);
      Serial.print(table[3][3]);
    }
    
    void setup()
    {
      Serial.begin(9600);
      int which = rand() % 2;
      myfunction(*tables[which]);
    }
    
    void loop()
    {}

  13. Bruce

    3 weeks ago

    When I use the variables from the Flash to call another function, funny things happen. For instance, this variant on the code you provide above, does not output correctly:
    #include <Flash.h>
    FLASH_TABLE(int, row_wave1, 4,
    {1,2,3,4}, {5,6,7,8}, {9,10,11,12}, {13,14,15,16});
    FLASH_TABLE(int, row_wave2, 4,
    {4,3,2,1}, {8,7,6,5}, {12,11,10,9}, {16,15,14,13});

    typedef _FLASH_TABLE TABLETYPE;
    TABLETYPE *tables[] = {&row_wave1, &row_wave2};

    void myfunction(TABLETYPE &table)
    {
    Serial.print(table[0][0]);
    Serial.print(table[1][1]);
    Serial.print(table[2][2]);
    Serial.print(table[3][3]);
    tablePrinter(table[0][0],table[1][1],table[2][2],table[3][3]);
    }

    void tablePrinter(int a, int b, int c, int d) {
    Serial.print(a);
    Serial.print(b);
    Serial.print(c);
    Serial.print(d);

    }
    void setup()
    {
    Serial.begin(115200);
    int which = rand() % 2;
    myfunction(*tables[which]);
    }

    void loop()
    {}

    the result is 4710134444
    Where the second sequence, ‘4444′ indicates that the same variable is being passed to ‘tablePrinter’ four times.

    I created this test code because I was finding the same effect in a larger project.


  14. Mikal

    3 weeks ago

    Most curious, Bruce. I get the same behavior when I test on my local box. I can’t explain it. In the _FLASH_ARRAY operator [], I was able to “correct” the behavior by adding the keyword “volatile” in a single line:

    return *reinterpret_cast<volatile T *>(&val);

    I’m not satisfied with this fix because I don’t understand why it works, but I don’t have time to delve into the assembly to find out right now. Thanks for the most interesting update.

    Mikal


  15. Matt E

    1 week ago

    Can Johann or Mikal elaborate a bit on this:

    @Matt E: You can add the following to Flash.h:

    // compare to another string
    bool operator==(const char *str)
    { return compare(str)==0; }

    // compare to anoter string: less, equal, greater
    inline byte compare(const char *str)
    { return -strcmp_P(str, _arr); }

    it allows comparisons:
    (myFlashStr == “abc”)

    I just downloaded Flash3. Where is the best place to add this to flash.h? I get the following errors when trying to compile my sketch.

    d:\Matt\Documents\Arduino\libraries\Flash\/Flash.h:43: error: ‘bool operator==(const char*)’ must have an argument of class or enumerated type

    d:\Matt\Documents\Arduino\libraries\Flash\/Flash.h: In function ‘byte compare(const char*)’:

    d:\Matt\Documents\Arduino\libraries\Flash\/Flash.h:48: error: ‘_arr’ was not declared in this scope


  16. jl

    6 days ago

    Hi!

    Thanx for this useful library. I’m just wondering, is it possible to create a flash array by “copying” a table. As earlier mentioned the values in a flash table can’t be edited afterwards so I thought about first putting the values in a normal array and then calling the FLASH_ARRAY from a function:

    int test[];

    … fill test[] with numbers…

    FLASH_ARRAY(int, flashArray, test);

    Doesn’t work. :(
    And this doesn’t seem to work either:

    int variable1=5;
    int variable2=10;

    FLASH_ARRAY(int, flashArray, variable1, variable2 etc.);

    It says: undefined reference to `__cxa_guard_acquire’

    Is it ONLY possible to create a flash array in this way:

    FLASH ARRAY(int, flashArray, 1,2,3,4,5, etc. the numbers one by one ); ?

    In my project I’m gonna have values coming in from a Processing sketch that I want to store in the Flash memory; there are a lot of values so the space in the EEPROM (512 bytes) won’t suffice. Do I have to start reading on the scary looking PROGMEM-stuff, or is it impossible even with PROGMEM?


  17. Mikal

    6 days ago

    jl– The good news for you is that you won’t have to learn all that scary-looking PROGMEM stuff. The bad news is, well, you guessed it: there is no way to take *run time* data and stuff it into flash memory. That’s only for constant data that is programmed at the time you upload your sketch.

    What you’re probably going to want is to interface with an external SD storage device. Here’s some discussion: http://www.arduino.cc/playground/Learning/SDMMC.

    Mikal

    Mikal


  18. jl

    5 days ago

    Thx for your answer.

2 Trackbacks For This Post
  1. New Flash library « Arduiniana

    [...] Interested to learn more? Read all about it here! [...]

  2. vkick Blog Site » Blog Archive » How to do Timer and Save Settings?

    [...] http://arduiniana.org/libraries/Flash/ Categories: Arduino Posted By: Mon Last Edit: 18 Jun 2009 @ 04 01 PM Email • Permalink Previous  Responses to this post » (None)  Post a Comment Click here to cancel reply. [...]

Leave a Reply