Skip to content

Port to Python 3.10

Sven Brauch requested to merge python310 into master

This is a somewhat working version which supports Python 3.10.

The general idea is that we somehow call ast.parse in Python and somehow transfer the information to C++. We can either do this call in-process, or in a separate process; the latter would require a client/server interface though because starting a process for each parse job is too slow. So I went for the first option.

My first attempt was serializing it to XML, then reading it out again; this works, but I now noticed it might also be feasible to use PyObject_GetAttrString-style introspection on the Python objects instead. We can still switch over to this. @flherne What do you think?

Performance shouldn't be affected much, since the whole parsing step is < 20% of effort anyways.

image

There are some issues remaining:

  • Some unit tests fail, these mostly seem to be somewhat recent changes to the AST, e.g. f(**{3:5}) doesn't correctly expand the double-starred dict.
  • Deserialization is missing some error handling, I think we want to enable exceptions and throw if the stream is ill-formed, then catch it and show an "internal parser error" or so.
  • Since this was a somewhat desparate attempt to get the project back up at all, I dropped support for Python < 3.10 for now. @flherne What do you think? Should we make an effort to support at least 3.9 and 3.8 or so?
  • I (intentionally) left out some cleanup of old AST nodes, maybe we need them for compatibility still. E.g. NumberAst is gone, it's replaced by ConstantAst.

There are some things which are not so nice but we probably can't do much:

  • The way KPluginLoader loads us as a plugin requires us to re-open libpython so that executed python code can find its symbols

On the positive side, I mostly got rid of our custom attribute range parsing by using the new endCol and endLine information.

Edited by Sven Brauch

Merge request reports