Support 3.12 feature: PEP 701 – Syntactic formalization of f-strings

[Spitfire1900]

The following valid Python samples from https://peps.python.org/pep-0701/ running under 3.12.0rc1 will cause exceptions in YAPF.

1. bag = {'wand': 'Elder'}
   print(f'Magic wand: { bag['wand'] }')

2. source = 'a_file.py'
   print(f"{source.removesuffix(".py")}.c: $(srcdir)/{source}")

3. print(f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}")

4. print(f"{'':*^{1:{1:{1}}}}")

5. x = 1
   y = 2
   print(
       f"___{
           x
       }___"
   )
   print(
       f"___{(
           y
       )}___"
   )

6. my_dict = {"key": "value"}
   print(f" something { my_dict["key"] } something else ")

[char101]

Quick hack just to make it work

1. Replace Grammar.txt with https://github.com/davidhalter/parso/blob/master/parso/python/grammar312.txt
2. Add the missing tokens

diff --git a/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py b/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py
index 0840c3c..b63fa46 100644
--- a/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py
+++ b/third_party/yapf_third_party/_ylib2to3/pgen2/grammar.py
@@ -179,6 +179,8 @@ opmap_raw = """
 //= DOUBLESLASHEQUAL
 -> RARROW
 := COLONEQUAL
+! FSTRINGCONVERSION
+... ELLIPSIS
 """

 opmap = {}
diff --git a/third_party/yapf_third_party/_ylib2to3/pgen2/token.py b/third_party/yapf_third_party/_ylib2to3/pgen2/token.py
index 5d09970..ee62bdb 100644
--- a/third_party/yapf_third_party/_ylib2to3/pgen2/token.py
+++ b/third_party/yapf_third_party/_ylib2to3/pgen2/token.py
@@ -66,6 +66,11 @@ ASYNC = 57
 ERRORTOKEN = 58
 COLONEQUAL = 59
 N_TOKENS = 60
+FSTRING_START = 61
+FSTRING_END = 62
+FSTRING_STRING = 63
+FSTRINGCONVERSION = 64
+ELLIPSIS = 65
 NT_OFFSET = 256
 # --end constants--

diff --git a/third_party/yapf_third_party/_ylib2to3/pygram.py b/third_party/yapf_third_party/_ylib2to3/pygram.py
index 4267c36..62714cb 100644
--- a/third_party/yapf_third_party/_ylib2to3/pygram.py
+++ b/third_party/yapf_third_party/_ylib2to3/pygram.py
@@ -31,10 +31,10 @@ python_grammar = driver.load_grammar(_GRAMMAR_FILE)
 python_symbols = Symbols(python_grammar)

 python_grammar_no_print_statement = python_grammar.copy()
-del python_grammar_no_print_statement.keywords['print']
+#del python_grammar_no_print_statement.keywords['print']

 python_grammar_no_print_and_exec_statement = python_grammar_no_print_statement.copy()  # yapf: disable # noqa: E501
-del python_grammar_no_print_and_exec_statement.keywords['exec']
+#del python_grammar_no_print_and_exec_statement.keywords['exec']

 pattern_grammar = driver.load_grammar(_PATTERN_GRAMMAR_FILE)
 pattern_symbols = Symbols(pattern_grammar)
diff --git a/yapf/pytree/pytree_utils.py b/yapf/pytree/pytree_utils.py
index e7aa6f5..9fb7433 100644
--- a/yapf/pytree/pytree_utils.py
+++ b/yapf/pytree/pytree_utils.py
@@ -87,7 +87,7 @@ def LastLeafNode(node):
 # Note that pygram.python_grammar_no_print_and_exec_statement with "_and_exec"
 # will require Python >=3.8.
 _PYTHON_GRAMMAR = pygram.python_grammar_no_print_statement.copy()
-del _PYTHON_GRAMMAR.keywords['exec']
+#del _PYTHON_GRAMMAR.keywords['exec']


 def ParseCodeToTree(code):

[Spitfire1900]

@char101 Current local testing causes a overwhelming majority of test cases to fail

[char101]

That's because the node names in Grammar.txt are different between the old and the parso's ones.

I have tried adding only the f-string nodes to the existing Grammar.txt but I couldn't make it work. Only by copying the whole Grammar.txt that it at least run without errors on the f-string example inputs.

[crankedguy]

is anyone on that right now? Just asking, I sadly lack the time to invest on my own while pulling up a company on the side... but it seemed like I waited 40 years for this feature to arrive in Python :) but cannot use yapf now anymore as a majority of refactored code wouldn't be formatted because of a few (or actually more than a few) lines of code...

[Spitfire1900]

@crankedguy
No one is handling this at this time. @char101 attempted to pull code from parso but it broke most unit tests.
It may be possible to create another set of backports from blib2to3 for this along with required app logic changes.

[crankedguy]

@Spitfire1900 ok thanks for the info

[char101]

There is already an implementation for black. Migrating the changes to lib2to3 is pretty easy, but the main code itself requires a lot of modification, which is not trivial.

Currently I'm using a black fork (cercis) which some changes that I prefer. What I dislike most about black is its excessive bracket splitting. In recent commit the official black also has a preview style of a more compact bracket splitting.