Port to Python 3.10
This is a somewhat working version which supports Python 3.10.
The general idea is that we somehow call ast.parse
in Python and somehow transfer the information to C++. We can either do this call in-process, or in a separate process; the latter would require a client/server interface though because starting a process for each parse job is too slow. So I went for the first option.
My first attempt was serializing it to XML, then reading it out again; this works, but I now noticed it might also be feasible to use PyObject_GetAttrString
-style introspection on the Python objects instead. We can still switch over to this. @flherne What do you think?
Performance shouldn't be affected much, since the whole parsing step is < 20% of effort anyways.
There are some issues remaining:
-
Some unit tests fail, these mostly seem to be somewhat recent changes to the AST, e.g. f(**{3:5})
doesn't correctly expand the double-starred dict. -
Deserialization is missing some error handling, I think we want to enable exceptions and throw if the stream is ill-formed, then catch it and show an "internal parser error" or so. -
Since this was a somewhat desparate attempt to get the project back up at all, I dropped support for Python < 3.10 for now. @flherne What do you think? Should we make an effort to support at least 3.9 and 3.8 or so? -
I (intentionally) left out some cleanup of old AST nodes, maybe we need them for compatibility still. E.g. NumberAst
is gone, it's replaced byConstantAst
.
There are some things which are not so nice but we probably can't do much:
- The way KPluginLoader loads us as a plugin requires us to re-open libpython so that executed python code can find its symbols
On the positive side, I mostly got rid of our custom attribute range parsing by using the new endCol
and endLine
information.