Skip to content

Add Python stub file generation with documentation parsing #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft

Add Python stub file generation with documentation parsing #388

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4537376
gen/gen_mpy: Add Python stub file generation for IDE support.
pi-anl Jun 9, 2025
f8918f4
build: Add LVGL JSON generation for enhanced bindings.
pi-anl Jul 17, 2025
0e8f6ea
gen: Implement separate stub generation system.
pi-anl Jul 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,17 @@ yacctab.py
*.bak
.history
.vscode

# Python cache files
__pycache__/
*.pyc
*.pyo
*.pyd

# Build artifacts
build/
*.pp
*.pp.c

# Test files
test_*/
129 changes: 126 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,8 +220,9 @@ Here is a procedure for adding lvgl to an existing MicroPython project. (The exa
### gen_mpy.py syntax
```
usage: gen_mpy.py [-h] [-I <Include Path>] [-D <Macro Name>]
[-E <Preprocessed File>] [-M <Module name string>]
[-MP <Prefix string>] [-MD <MetaData File Name>]
[-E <Preprocessed File>] [-J <JSON file>]
[-M <Module name string>] [-MP <Prefix string>]
[-MD <MetaData File Name>]
input [input ...]

positional arguments:
Expand All @@ -236,6 +237,9 @@ optional arguments:
-E <Preprocessed File>, --external-preprocessing <Preprocessed File>
Prevent preprocessing. Assume input file is already
preprocessed
-J <JSON file>, --lvgl-json <JSON file>
Provde a JSON from the LVGL JSON generator for missing
information
-M <Module name string>, --module_name <Module name string>
Module name
-MP <Prefix string>, --module_prefix <Prefix string>
Expand All @@ -244,12 +248,38 @@ optional arguments:
Optional file to emit metadata (introspection)
```

Example:
Example for gen_mpy.py:

```
python gen_mpy.py -MD lv_mpy_example.json -M lvgl -MP lv -I../../berkeley-db-1.xx/PORT/include -I../../lv_binding_micropython -I. -I../.. -Ibuild -I../../mp-readline -I ../../lv_binding_micropython/pycparser/utils/fake_libc_include ../../lv_binding_micropython/lvgl/lvgl.h
```


### gen_stubs.py syntax
```
usage: gen_stubs.py [-h] --metadata METADATA --stubs-dir STUBS_DIR
[--lvgl-dir LVGL_DIR] [--module-name MODULE_NAME]
[--validate]

Generate LVGL Python stub files

options:
-h, --help show this help message and exit
--metadata METADATA JSON metadata file from gen_mpy.py
--stubs-dir STUBS_DIR
Output directory for stub files
--lvgl-dir LVGL_DIR LVGL source directory for documentation
--module-name MODULE_NAME
Module name
--validate Enable validation checks
```

Example for gen_stubs.py:

```
python gen_stubs.py --metadata lv_mpy_example.json --stubs-dir ./stubs-output --lvgl-dir ../lvgl --module-name lvgl --validate
```

### Binding other C libraries

The lvgl binding script can be used to bind other C libraries to MicroPython.
Expand Down Expand Up @@ -507,3 +537,96 @@ print('\n'.join(dir(lvgl.btn)))
...
```

## IDE Support and Type Stubs

For better development experience with IDEs (VS Code, PyCharm, etc.), this repository includes Python type stubs that provide autocompletion, type checking, and documentation hints.

### Installation

The type stubs are automatically generated during the build process and packaged for easy installation:

#### Development Installation (recommended)

For development with the latest stubs:

```bash
pip install -e /path/to/lv_binding_micropython/stubs
```

#### From Built Package

After building and packaging:

```bash
pip install lvgl_stubs
```

### Features

Once installed, your IDE will automatically provide:

- **Autocompletion** for all LVGL functions, methods, and properties
- **Type checking** with mypy and other type checkers
- **Function signatures** with parameter names and types
- **Documentation strings** extracted from LVGL C headers
- **Source location references** to help navigate LVGL documentation

### Building Stubs

The stubs are generated using a separate `gen_stubs.py` module that creates both stub files and distributable wheel packages. Two make targets are available:

```bash
# Generate stub files only
make USER_C_MODULES=/path/to/lv_binding_micropython LVGL_STUBS

# Generate stub files and build distributable wheel package
make USER_C_MODULES=/path/to/lv_binding_micropython LVGL_STUBS_WHEEL
```

You can also generate stubs manually:

```bash
cd gen
python gen_stubs.py --metadata /path/to/metadata.json --stubs-dir /path/to/output/dir --lvgl-dir /path/to/lvgl --module-name lvgl --validate
```

### Build Process

The stub package generation uses a template-based approach:

1. **Template Folder**: The `./stubs/` directory contains a complete Python package template with:
- `pyproject.toml` - Package configuration
- `lvgl_stubs/` - Python package directory
- `lvgl_stubs/__init__.py` - Package initialization
- `lvgl_stubs/py.typed` - Marks package as typed

2. **Build Process**: During build, the template is copied to the build directory, then:
- LVGL version is extracted from headers (`lv_version.h`)
- `__version__.py` file is created with the extracted version
- `lvgl.pyi` stub file is generated with all type definitions
- Documentation is extracted from 200+ LVGL header files in parallel
- Validation ensures stubs match the generated MicroPython API

3. **Package Creation**: The build process creates a distributable wheel package (`lvgl_stubs-X.Y.Z-py3-none-any.whl`) that can be installed with pip.

### Generated Content

The generated `lvgl.pyi` stub file contains type definitions for:
- All LVGL widget classes (buttons, labels, sliders, etc.)
- Module-level functions and constants
- Enum definitions with proper typing
- Struct types with documented fields
- Callback function signatures

Documentation is automatically extracted from LVGL C headers using Doxygen-style comment parsing, providing rich IDE tooltips and help text.

### Package Structure

The stubs are packaged in `stubs/` template directory with:
- `pyproject.toml` - Package configuration with dynamic versioning
- `lvgl_stubs/` - Python package containing type stubs
- `lvgl_stubs/__init__.py` - Package initialization with version import
- `lvgl_stubs/py.typed` - Marks package as typed
- `lvgl_stubs/__version__.py` - Version file (generated during build)
- `lvgl_stubs/lvgl.pyi` - Generated type stubs (created during build)

22 changes: 16 additions & 6 deletions gen/gen_mpy.py
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please move all stubs generation implementation into a separate gen_stubs.py module?
Then call it from this gen_mpy.py.
Just to make code cleaner, and code/stubs generators separate.

Copy link
Collaborator

@PGNetHun PGNetHun Jul 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But if your stubs generation implementation can run independent from gen_mpy.py (because no gen_mpy functions are used), then I vote for having gen_stubs.py file as a separate runnable script, and call it from makefile.
So, basically separate gen_mpy.py and gen_stubs.py

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea. I'd initially thought the stubs would be easier to generate as the functions are being parsed but yeah it ended up working better as a separate pass looking for different details. It should be straightforward now to split it out into its own file and I'll see how it goes making it standalone!

Original file line number Diff line number Diff line change
Expand Up @@ -35,13 +35,22 @@ def eprint(*args, **kwargs):
from argparse import ArgumentParser
import subprocess
import re
import os
import textwrap
from os.path import dirname, abspath
from os.path import commonprefix

script_path = dirname(abspath(__file__))
sys.path.insert(0, "%s/../pycparser" % script_path)
from pycparser import c_parser, c_ast, c_generator

#
# Constants
#

# Import shared utilities
from gen_utils import simplify_identifier, get_enum_name, c_type_to_python_type, LVGL_DOC_SEARCH_DIRS

#
# Argument parsing
#
Expand Down Expand Up @@ -103,6 +112,7 @@ def eprint(*args, **kwargs):
metavar="<MetaData File Name>",
action="store",
)
# Stub generation moved to separate gen_stubs.py script
argParser.add_argument("input", nargs="+")
argParser.set_defaults(include=[], define=[], ep=None, json=None, input=[])
args = argParser.parse_args()
Expand Down Expand Up @@ -354,9 +364,7 @@ def sanitize(


@memoize
def simplify_identifier(id):
match_result = lv_func_pattern.match(id)
return match_result.group(1) if match_result else id
# simplify_identifier is now imported from gen_utils


def obj_name_from_ext_name(ext_name):
Expand All @@ -382,9 +390,7 @@ def method_name_from_func_name(func_name):
return res if res != "del" else "delete" # del is a reserved name, don't use it


def get_enum_name(enum):
match_result = lv_enum_name_pattern.match(enum)
return match_result.group(3) if match_result else enum
# get_enum_name is now imported from gen_utils


def str_enum_to_str(str_enum):
Expand Down Expand Up @@ -3796,6 +3802,8 @@ def generate_struct_functions(struct_list):
)
)

# Python stub file generation functions moved to gen_stubs.py

# Save Metadata File, if specified.

if args.metadata:
Expand Down Expand Up @@ -3829,3 +3837,5 @@ def generate_struct_functions(struct_list):

with open(args.metadata, "w") as metadata_file:
json.dump(metadata, metadata_file, indent=4)

# Python stub generation has been moved to separate gen_stubs.py script
Loading
Loading