CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model

Yuxuan Luo, Jiaqi Tang, Chenyi Huang, Feiyang Hao, Zhouhui Lian*

• Task: Create a versatile VLM for Chinese calligraphy contextualization ($CC^2$), including full-page/region-wise recognition, multilingual interpretation, and intent analysis.
• Ideas: Proposed character-wise slicing to simplify recognition, introduced the CalliAlign module for embedding alignment, developed Embedding Instruction Tuning (e-IT) to enhance interpretation, and created CalliBench — the first comprehensive benchmark for full-page calligraphic contextualization.
• Job: Conducted a literature review of OCR methods and reasoning benchmarks for VLMs, performed comparisons between our model, state-of-the-art (SOTA) OCR methods, and other competitive VLMs, and participated in benchmark construction, model fine-tuning, ablation study, and paper writing.

Open-ended Financial Competency Assessment

Keywords: Conditional Semantic Textual Similarity (C-STS), Prompt Engineering, Active Learning, LLMs

• Task: Develop an LLM-based evaluation framework to address efficiency and fairness challenges in open-ended financial competency assessments for recruitment and customer capability evaluations.
• Ideas: Designed domain-specific open-ended questions and a multi-dimensional C-STS assessment framework, generated high-quality datasets, and optimized model performance using active learning techniques.
• Job: Built a multi-dimensional (knowledge, behavior, awareness) financial QA dataset via Chain of Thought (CoT) / In Context Learning (ICL) / role-playing prompting, and conducted literature reviews.

IntE: Unstructured Society Research Interview Data Evaluation Method Based on LLM Powered Conditional Semantic Text Similarity

Keywords: Post-training, Human-Computer Interaction (HCI), LLMs, UI Design

• Task: Create a phased evaluation system to address credibility gaps in open-ended survey results for HCI research.
• Ideas: Decomposed the process into stages such as prompt refinement, answer generation, analysis, and credibility assessment, leveraging role-specific LLM collaboration and researcher-centric UI design.
• Job: Contributed to dataset development and ablation studies while designing an intuitive user interface that enables non-technical researchers to interact effortlessly with the complex multi-agent system, and implementing with PyQt.

CalliArranger: Input-Adaptive Bounding Box Generation for Layout-Specific Calligraphy Arrangement

Keywords: LoRA, q-LoRA, Visualization

• Task: Develop an efficient calligraphy layout generation solution to overcome cost and quality limitations of traditional methods using LLMs and prompt engineering.
• Ideas: Curated a calligraphy image dataset, evaluated baseline methods, and implemented RLHF with visualization-based validation.
• Job: Selected a base model based on data distribution, performance, and computational efficiency, fine-tuned using LoRA/q-LoRA methods and evaluated through visualization techniques, which achieved higher quality results while significantly reducing costs compared to traditional methods.

Multithreaded Optimized Terminal AI Chatbot

Keywords: Multithreading, Thread Pool, System Signals, Conditional Variables, Semaphores

• Developed a terminal-based AI chatbot using multithreading techniques, implementing thread pools to accelerate tensor calculations for model inference and using another thread to handle user interface.
• Synchronized threads using system signals, conditional variables, and semaphores, and analyzed efficiency under varying thread counts and scheduling policies.

PokeFusionLab: Fine-tune a Diffusion Model with Various Datasets, Co-Leader

Keywords: Latent Diffusion Models, LoRA, Data-Centric AI

• Fine-tuned latent diffusion models using Low-Rank Adaptation (LoRA) to generate images in specific styles, experimenting with single-image and Pokémon dataset training.
• Utilized Qwen2-VL-7b to re-annotate datasets with varying caption detail levels, fine-tuning models to analyze the impact of detailed captions on text-to-image generation performance.

VR Music Firework Show

Keywords: Unity 3D, Google Cardboard SDK

• Created an immersive VR experience allowing users to enjoy fireworks with beautiful music across multiple scenic backgrounds.

Dormitory Management System, Leader

Keywords: Vue.js, Spring Boot, MyBatis

• Developed a dormitory management system with data panels that allows users to manage with multi-granularity.