Shopify Develops LLM Prompt Compression Technology 'Gisting'

Shopify's engineering team has developed 'Gisting,' a technology that compresses long system prompts for large language models (LLMs). This approach replaces lengthy prompts with a small number of learned tokens called 'gist tokens,' reducing the cost of inference while improving processing throughput (the amount of processing per unit time).

When integrating LLMs into real-world services, a 'system prompt' that defines model behavior is essential. This prompt is text that conveys to the AI its role, constraints, and response policies, and tends to become longer for complex services. Meanwhile, because LLMs process input text on a token-by-token basis, longer prompts increase computational costs and affect response speed. Against this backdrop, interest in methods to optimize prompts has grown across the industry.

The approach adopted by Gisting involves 'compressing' the content of a prompt into a small number of meaningful tokens. Specifically, instead of passing a long system prompt to the model each time as-is, it is converted beforehand into a small group of tokens that have learned its content. During inference, using these compressed tokens makes it possible to significantly reduce the total number of tokens in the input.

The significance of this technology extends beyond cost reduction. In services using LLMs, inference costs represent a major portion of operational expenses. By reducing token count through prompt compression, computational requirements per request decrease, allowing the same computational resources to handle more requests. For e-commerce platforms handling large volumes of requests in particular, such efficiency improvements represent substantial economic impact.

Shopify operates services leveraging LLMs, and the motivation for its engineering team to address this challenge is clear. Gisting, as a technology born from real operational challenges, possesses practical value that goes beyond mere academic proposals. It can serve as a reference approach for other companies and developers facing similar challenges.

The field of prompt compression is expected to become increasingly important as LLM adoption expands. Separate from model evolution itself, answers are needed to the question of how to perform inference more efficiently, and approaches like Gisting point to one such direction. Inference cost optimization is an unavoidable theme for sustainable AI technology operations, and similar initiatives from various companies are likely to emerge in the future.