linear programming

ConvexTok Reframes Tokenization as Optimization

ConvexTok turns tokeniser construction into a linear program and gets closer-to-optimal tokenization.