Fix issues due to breaking tokenize changes in 3.12 (#14107)...
Fix issues due to breaking tokenize changes in 3.12 (#14107)
Hey 👋
I'm opening this PR to start a discussion on which of these fixes are
valid or not. I may have cut some corners in some places, mostly because
of lack of knowledge or context.
After we've fixed the issues and CI is green, we can have a bigger
discussion on whether to go foward with the `tokenize` module or some
other tokenizer.