MicroPython Native Code Emitter Gotchas

MicroPython Native Code Emitter Gotchas - Pozetron Inc

This guest post was written by the terrifically talented engineer Alexander Lukanin.

The Limitations and Pitfalls of `@micropython.native`

Micropython is fully compatible with Python 3 syntax… except when it isn’t.

In this article, we explore one of those “dark corners” of micropython: the limitations imposed by the @micropython.native decorator.

Documentation on The Native code emitter is brief and a bit vague:

Context managers are not supported (the with statement).

Generators are not supported.

If raise is used an argument must be supplied.

So, what exactly works and what fails?
Are there any differences in how Micropython v1.9.3 behaves on Unix and on ESP8266?
Let’s find out!

Generators don’t work with `@micropython.native`

This is the easiest one: yield and @micropython.native just don’t mix.

    >>> @micropython.native
    ... def native_yield():
    ...         yield 1
    ...
    NotImplementedError: native yield

It’s the same error on Unix and on the ESP8266.

What if we call another generator from a native function?

    >>> def generate():
    ...     yield 1
    ...
    >>> @micropython.native
    ... def native_call_yield():
    ...     for x in generate():
    ...         return x
    ...
    >>> native_call_yield()
    1

Looks like Micropython has no problem with that.

`except`, `raise` and `@micropython.native`

From the documentation, it may seem that raise without an argument will just fail at compile time with NotImplementedError, the same way as yield. But the actual behavior is more complex.

An explicit exception name works fine, of course:

    @micropython.native
    def native_ex_raise():
        try:
            ...
        except SomeException as ex:
            # do something...
            raise ex

To much surprise, the implicit except/raise also seems to work, at least in our minimal example:

    @micropython.native
    def native_no_ex_no_raise():
        try:
            ...
        except SomeException:
            # do something...
            raise

This is a popular Python idiom, for two reasons:

In Python 2, raise without argument preserves the original stack trace (raise ex doesn’t).
It’s clear and concise.

But should we use it in Micropython? Probably not, pfalcon himself said that “except” without “as” is strongly discouraged.

Now, let’s try to mix except ... as ... and raise. It’s perfectly legal in “normal” Python, but in Micropython with the Native code emitter it fails; moreover, it is inconsistent across architectures.

Unix

    >>> @micropython.native
    ... def native_ex_no_raise():
    ...     try:
    ...         raise SomeException('BOOM!')
    ...     except SomeException as ex:
    ...         print('catch: native_ex_no_raise')
    ...         raise
    ...
    >>> native_ex_no_raise()
    catch: native_ex_no_raise
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: exceptions must derive from BaseException

ESP8266

    >>> @micropython.native
    ... def native_ex_no_raise():
    ...     try:
    ...         raise SomeException('BOOM!')
    ...     except SomeException as ex:
    ...         print('catch: native_ex_no_raise')
    ...         raise
    ...
    MemoryError: memory allocation failed, allocating 368 bytes for native code

`@micropython.native` + custom exceptions == weirdness

Working with exceptions in Native functions may generally be a bad idea.

    >>> with open('weird.py', 'w') as f:
    ...     f.write('import micropython\n')
    ...     f.write('class SomeException(Exception):\n')
    ...     f.write('  pass\n')
    ...     f.write('@micropython.native\n')
    ...     f.write('def raise_some():\n')
    ...     f.write('  raise SomeException\n')
    ...     f.write('raise_some()\n')
    ...
    >>> import weird
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "weird.py", line 7, in <module>
    SomeException:

Everythings is fine so far.

    >>> weird.raise_some()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'weird' is not defined

OK, the module was not imported because an exception was raised. But, surprisingly, it imports successfully on the second try (in CPython, it would fail every time):

    >>> import weird
    >>> weird.SomeException
    <class 'SomeException'>
    >>> weird.raise_some
    <function>

Can we call the function now?

    >>> weird.raise_some()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'SomeException' is not defined

At first glance, it looks like a programmer’s error, but it’s not; and the same code without @micropython.native works fine. Apparently, the Native code emitter has some additional undocumented restrictions. This deserves further investigation.

`@micropython.native` and context managers

One more pitfall. What works on Unix…

    >>> @micropython.native
    ... def native_with():
    ...     with open('a.txt', 'w') as file:
    ...         file.write('Hello world')
    ...
    >>> native_with()
    >>> print(open('a.txt').read())
    Hello world

…fails on the ESP8266:

    >>> @micropython.native
    ... def native_with():
    ...     with open('a.txt', 'w') as file:
    ...         file.write('Hello world')
    ...
    MemoryError: memory allocation failed, allocating 452 bytes for native code

Maybe open is special? Nope, if you write a custom context manager, the behavior is the same: works on Unix, fails on ESP8266.

Finally, just for fun. Sometimes Micropython just explodes:

  >>> native_ex_no_raise()
  catch: native_ex_no_raise
  Fatal exception 28(LoadProhibitedCause):
  epc1=0x4024d147, epc2=0x00000000, epc3=0x00000000, excvaddr=0x01640064, depc=0x00000000

   ets Jan  8 2013,rst cause:2, boot mode:(3,6)

  load 0x40100000, len 31096, room 16
  tail 8
  chksum 0x22
  load 0x3ffe8000, len 1084, room 0
  tail 12
  chksum 0xd7
  ho 0 tail 12 room 4
  load 0x3ffe8440, len 3252, room 12
  tail 8
  chksum 0xa3
  csum 0xa3

Conclusion

In some cases, Micropython behaves differently between the Unix and ESP8266 ports.
Use except SomeException as ex: and raise ex. Don’t use except SomeException: and raise without an argument.
Be careful with @micropython.native because the trade-off for the improved performance isn’t just an increase in compiled code size.