Encountered the same issue with cx_freeze. Setting 'build_exe' option 'optimize' to 0 or 1 is a viable workaround. Also adding:

    if dispatcher.__doc__ is None:
        dispatcher.__doc__ = ""

to the beginning of array_function_dispatch function in numpy.core.overrides works as well.

Neither are ideal, but they work if/until this is addressed

Thanks for the update @kg-2 , are you also using the same OS & Python version?

I hope that @shoyer will check this issue.

I have to say I'm rather puzzled here. We do have integration tests that verify NumPy works when run with PYTHONOPTIMIZE=2, and add_docstring will bail out even if docstrings are undefined even before parsing its arguments:

Is it possible that these tools don't set Py_OptimizeFlag properly at runtime?

Can you verify that you can successfully run your example with NumPy 1.15?

@shoyer:

I have to say I'm rather puzzled here.

Also here, will try to build and run the tests.
Meanwhile can you explain why docs_from_dispatcher=True is True by default?

Thanks for the update @kg-2 , are you also using the same OS & Python version?

I hope that @shoyer will check this issue.

Same numpy version (1.16.2), python 3.6.8 stock build. Something changed starting with numpy 1.13 that broke things for me.
**Edit: I'm also on Windows 10

Does not reproduce with cpython3.7, pyinstaller3.4, numpy1.16.2 on ubuntu 18.04. Also tried with cpython3.6 and the example works fine.

Meanwhile can you explain why docs_from_dispatcher=True is True by default?

Because we do want to copy docstrings from dispatchers when using array_function_from_dispatch (at least, as long as Python is not run with -OO). The implementation argument in these cases is typically a function written in C (and thus does not have an original docstring).

After some deeper checking, building and comparing with NumPy 1.15 build it seems that PyInstaller passes the Py_OptimizeFlag correctly when it freeze the code.

BUT the output bundle do not store the flag state when it was generated.
The generated bundle will have the sys.frozen=True attribute with sys.flags.optimize=0 this is why Py_RETURN_NONE is not returned as Py_OptimizeFlag state was not saved.

As i see it (i may be wrong) array_function_from_dispatch may be called when no docstrings are available such the above case, thus it breaks the docs_from_dispatcher=True logic.

i'm not sure how to continue from here, is this something that needed to be handled in NumPy or PyInstaller?

For my scenario this next code did the job without doing changes to NumPy core files:

# added in the main module before import numpy
import sys
if __doc__ is None and hasattr(sys, 'frozen'):
    from ctypes import c_int, pythonapi
    c_int.in_dll(pythonapi, 'Py_OptimizeFlag').value = 2

or more generic:

def frozen_oo():
    """Check if code is frozen with optimization=2"""
    import sys
    if frozen_oo.__doc__ is None and hasattr(sys, 'frozen'):
        from ctypes import c_int, pythonapi
        c_int.in_dll(pythonapi, 'Py_OptimizeFlag').value = 2

frozen_oo()
import numpy as np
...

I think this is most likely a bug in PyInstaller

and cx_freeze

And Buildozer when making Android package

I am running into the same issue using WiX Toolset to create Windows installers when using the -OO flag.

I encountered this issue using cx_Freeze. What happens is that cx_Freeze packages .pyc-files only, not including the original .py-files. If the .pyc-files are created using -OO, but are run without -OO, then this is what happens: Docstrings are gone, but the flag that tells Python to remove the docstrings is no longer set.

This can be reproduced without either of cx_Freeze, PyInstaller or WiX toolset, by replacing (e.g.) multiarray.py in numpy.core with a file multiarray.pyc that is a copy of multiarray.cpython-38.opt-2.pyc.

Whether you see this as is a bug in the freezers or in numpy is a matter of taste; you can argue for both. Python has always supported using .pyc-files in place of .py-files, but that doesn't mean numpy necessarily has to support this aspect of Python. I would appreciate if it did, though.

Skimming https://www.python.org/dev/peps/pep-0488/, it seems that .pyc files are guaranteed to not be optimized. So the case when "the .pyc-files are created using -OO" is illegal:

A PYC file is the bytecode file generated and read from when no optimization level is specified at interpreter startup (i.e., -O is not specified).

Optimized pyc files must be saved with a .cpython-35.opt-2.pyc suffix.

So this is a bug in the freezer.

You are quoting a paragraph about the state of affairs prior to PEP 488. Nowadays, all compiled bytecode files are called .pyc, regardless of optimisation level. See the section https://www.python.org/dev/peps/pep-0488/#compatibility-considerations.

The interpreter does not recognise cache file names like multiarray.cpython-38.opt-2.pyc on the source path, only multiarray.pyc.

Not as I read it - the state of affairs prior to PEP 488 was to have .pyc and .pyo files; the state of affairs after PEP 488 is to have .cpython-35.pyc, .cpython-35.opt-1.pyc , and .cpython-35.opt-2.pyc files.

The interpreter does not recognise cache file names like multiarray.cpython-38.opt-2.pyc on the source path

Correct, and this is a feature. The interpreter only recognizes these files if launched with -OO, just as a python3.7 interpreter will not recognized .cpython-38.* pyc files.

The interpreter recognises neither of .cpython-38.opt-1.pyc, .cpython-38.opt-2.pyc or .cpython-38.pyc on the source path. These are for the cache directory only. Bytecode files in a bytecode-only deployment are called .pyc, regardless of optimisation level.

Apologies, your link above was broken and I didn't notice. I've edited it to work, and read it now. Is there anywhere I can read up on bytecode-only deployments? I assume you can't just distribute a __cache__ folder with no python files alongside it?

Ah, PEP 3147 has this to say:

If the py source file is missing, the pyc file inside __pycache__ will be ignored. This eliminates the problem of accidental stale pyc file imports.

For backward compatibility, Python will still support pyc-only distributions, however it will only do so when the pyc file lives in the directory where the py file would have been, i.e. not in the __pycache__ directory. pyc file outside of pycache will only be imported if the py source file is missing.

Presumably this whole problem can be reduced to:

# foo.pyc
""" this docstring may be optimized out"""

# bar.py
import sys
import foo
if sys.flags.optimize < 2:
    assert foo.__doc__

and compiling / running with different optimization levels. If so, I suggest you open an issue with PyInstaller asking what the expected behavior is and why.

Exactly. Numpy makes the assumption that all bytecode executed has been compiled with the current value of sys.flags.optimize. When this assumption does not hold, numpy fails.

CPython supports mixed-optimisation execution, numpy does not.

There are no special rules for bytecode-only deployments. Any module can be implemented with a .pyc instead of a .py, you just put the .pyc where the .py would have been. If you just happen to implement all modules that way, then the deployment is bytecode-only. Freezers combine that with zipimport to create a packaging that may look very distinctive and alien, but it's really just a combination of two standard features, .pyc-files on the source path and zipimport.

CPython supports mixed-optimisation execution

I'm not at all convinced this is deliberate. Here's what I consider the minimal case:

"""
This file will succeed if imported from a .opt-n.pyc file in __pycache__, but
fail if run from a manually created `.pyc` compiled at a different optimization
level to the interpreter that runs it. Is this second use-case an invalid use of pyc files?
"""
import sys

if __doc__:
    assert sys.flags.optimize in (0, 1)
else:
    assert sys.flags.optimize == 2

if __debug__:
    assert sys.flags.optimize == 0
else:
    assert sys.flags.optimize in (1, 2)

The question is, does cpython consider use-cases when this program fails valid? @brettcannon, as the author of PEP 488, do you have any thoughts here, or suggestions of a better place to direct this question?

Do we have any reason to not just make a None input a no-op in add_docstring? The code is here:

Although it is unclear to me whether some related tests might fail in such an environment (whether that should exist or not).

EDIT: The argument parsing code is just below that

CPython assumes that the bytecode you are importing was compiled at the same level as the interpreter running. Mixing stuff is just asking for trouble.

Mixing stuff is just asking for trouble.

Of course. But it has never been disallowed that I know of, it's been possible to do for at least two decades. Maybe it should be explicitly disallowed, with the interpreter rejecting bytecode with the wrong flag set internally. That certainly would put the pressure on to fix marcelotduarte/cx_Freeze#84 and related issues.

Note that cx_Freeze doesn't actually mix stuff - all the .pyc's are consistently optimised to the same level, it's just that the stub program hardwires Py_OptimizeFlag==0. Why that's seemingly so hard to change I don't know.

@AndersMunch that would require a change to the .pyc file format as the "compiler" flags are not stored as part of the bytecode itself. So you could still "lie" if you renamed a file.

Honestly, this is the first time anyone has asked me since I wrote importlib back when I was still working on my PhD about this, so I don't think it's worth the hassle to explicitly prevent this. Python is a language of consenting adults, so my answer is simply, "don't do that if you know what's good for you" 😉 .

Encountered the same issue with cx_freeze. Setting 'build_exe' option 'optimize' to 0 or 1 is a viable workaround. Also adding:

    if dispatcher.__doc__ is None:
        dispatcher.__doc__ = ""

to the beginning of array_function_dispatch function in numpy.core.overrides works as well.

Neither are ideal, but they work if/until this is addressed

Still we need to this manual work around :-(

My temporary solution is:

pip install 'numpy<1.16'

Still an issue with CPython 3.10.8, cx-Freeze 6.12.0, numpy 1.23.4 on Windows 10 amd64.

Issue #10167 proposes to refactor docstrings and add_docstrings. The proposal is to move docstrings for C functions into C code, which would avoid this problem altogether. Barring that, I think the suggestions to avoid raising an error on missing docstrings is the best work around.

As @brettcannon indicated, the issue is related to compiling at one optimization and running the interpreter at another. A simple solution on the PyInstaller side is to ensure both Analysis and EXE are given the same optimization level. If you are running Pyinstaller on the command line, you would use the --optimize 2 flag described here:
https://pyinstaller.org/en/stable/usage.html#cmdoption-optimize

If you are using a spec file, it would look something like this, noting that ('O', None, 'OPTION') is given twice to EXE in the options:

a = Analysis(
    ['example.py'],
    pathex=[],
    binaries=[],
    datas=[],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    noarchive=False,
    optimize=2,
)
pyz = PYZ(a.pure)

exe = EXE(
    pyz,
    a.scripts,
    [('O', None, 'OPTION'), ('O', None, 'OPTION')],
    exclude_binaries=True,
    name='example',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None,
)