Projects

Logic Before Language pipeline

Proposed logic pre-pretraining (Logic-PPT), which uses formal derivations to provide language models with richer structural and linguistic biases before natural language training.
Demonstrated faster skill acquisition at the 100B-token scale, reaching 80% accuracy on linguistic tasks with 36B fewer training tokens than standard initialization.
Showed that logic pre-pretraining induces a lower-rank, spectrally concentrated representation space and improves model compressibility, matching dense baseline performance at approximately 33% sparsity.

GeoUni Overview

Proposed the first unified multi-modal geometry expert model, GeoUni, capable of solving geometry problems, generating precise geometric diagrams using both formal and natural language, and creating geometry problems based on knowledge points.
Proposed Geo-MAGVIT, a module specifically designed for the tokenization of geometric diagrams. By introducing topo-structural awareness loss and text region loss, it significantly improves the precision of geometry structure and text reconstruction.
Combined GRPO and LoRA to train the Geo-Reasoning-Adapter, which effectively boosts geometric reasoning capability and seamlessly integrates into the unified model architecture.
Established novel diagram generation evaluation metrics, including the Geometry Semantic Matching Scores (GSMSs) and Geometry Pixel Matching Score (GPMS), to comprehensively evaluate the diagram generation task.

Diagram Formalization Enhanced Geometry Problem Solver pipeline

Designed a multimodal framework integrating visual features and geometric formal languages for solving complex geometry problems.
Proposed a synthetic data approach (SynthGeo228K dataset) for improving model training and diagram interpretation.
Achieved an accuracy of 82.38% on the publicly available FormalGeo7k dataset, significantly outperforming existing multi-modal and language models, including GPT-4.