diff --git a/docs/README.md b/docs/README.md index df486afbaa..3b81f8b034 100644 --- a/docs/README.md +++ b/docs/README.md @@ -72,4 +72,14 @@ This guide is for contributors looking to make changes to the documentation in t 2. **Push your new changes on a new branch**: Feel free to add or edit existing documentation and open a PR for your changes. Once your PR is reviewed and approved, the changes will be ready to merge into main. -3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins. \ No newline at end of file +3. **Updating the website**: Once your changes are merged to main, the changes would be reflected on live websites usually in 5-15 mins. + +## LLMs.txt Files + +DSPy documentation automatically generates [llmstxt.org](https://llmstxt.org/) compliant files to help LLMs understand and work with DSPy: + +- **`/llms.txt`** - A structured overview with links to key documentation sections +- **`/llms-full.txt`** - A comprehensive version with full content for deep understanding + +These txt files are automatically generated during the docs build process using the [mkdocs-llmstxt](https://github.com/pawamoy/mkdocs-llmstxt) plugin. When modifying documentation, consider updating the `llmstxt` plugin section within `mkdocs.yml` to help LLMs better understand DSPy's capabilities. + diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 9d9c90f6eb..88821dd2cb 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -236,6 +236,82 @@ plugins: "docs/quick-start/getting-started-02.md": "tutorials/rag/index.ipynb" "quick-start/getting-started-01.md": "tutorials/rag/index.ipynb" "quick-start/getting-started-02.md": "tutorials/rag/index.ipynb" + - llmstxt: + markdown_description: > + DSPy is the framework for programming—rather than prompting—language models. + DSPy unifies techniques for prompting, fine-tuning, reasoning, tool use, and evaluation of LMs. + It provides a systematic approach to building AI applications through composable modules, + optimization techniques, and evaluation frameworks. + full_output: llms-full.txt + sections: + Getting Started: + - index.md: DSPy overview and quick start + - faqs.md: Frequently asked questions + - cheatsheet.md: DSPy cheatsheet + Core Learning: + - learn/index.md: Learn DSPy fundamentals + - learn/programming/*.md: Programming concepts - signatures, modules, language models + - learn/evaluation/*.md: Evaluation framework - data handling, metrics, overview + - learn/optimization/*.md: Optimization techniques and optimizers + Building AI Programs: + - tutorials/build_ai_program/index.md: Building AI programs overview + - tutorials/conversation_history/index.md: Managing conversation history + - tutorials/custom_module/index.ipynb: Customizing DSPy modules + - tutorials/classification/index.md: Classification with DSPy + - tutorials/rag/index.ipynb: Retrieval-Augmented Generation + - tutorials/agents/index.ipynb: Building AI agents + - tutorials/customer_service_agent/index.ipynb: Customer service agents + - tutorials/entity_extraction/index.ipynb: Entity extraction + - tutorials/multihop_search/index.ipynb: Multi-hop search and reasoning + - tutorials/program_of_thought/index.ipynb: Program of thought reasoning + - tutorials/tool_use/index.ipynb: Advanced tool use + Optimization and Training: + - tutorials/optimize_ai_program/index.md: Optimization overview + - tutorials/math/index.ipynb: Math reasoning optimization + - tutorials/classification_finetuning/index.ipynb: Classification fine-tuning + - tutorials/games/index.ipynb: Agent fine-tuning with games + - tutorials/rl_ai_program/index.md: Reinforcement learning overview + - tutorials/rl_papillon/index.ipynb: RL for privacy-conscious delegation + - tutorials/rl_multihop/index.ipynb: RL for multi-hop research + Specialized Use Cases: + - tutorials/papillon/index.md: Privacy-conscious delegation + - tutorials/image_generation_prompting/index.ipynb: Image generation prompting + - tutorials/audio/index.ipynb: Audio processing with DSPy + - tutorials/ai_text_game/index.md: Creative text-based AI games + - tutorials/email_extraction/index.md: Email information extraction + - tutorials/sample_code_generation/index.md: Code generation for libraries + - tutorials/llms_txt_generation/index.md: Generating llms.txt files + - tutorials/mem0_react_agent/index.md: Memory-enabled ReAct agents + - tutorials/yahoo_finance_react/index.md: Financial analysis agents + Development and Deployment: + - tutorials/core_development/index.md: Development workflow + - tutorials/mcp/index.md: Model Context Protocol integration + - tutorials/output_refinement/best-of-n-and-refine.md: Output refinement techniques + - tutorials/saving/index.md: Saving and loading models + - tutorials/cache/index.md: Caching strategies + - tutorials/deployment/index.md: Production deployment + - tutorials/observability/index.md: Debugging and observability + - tutorials/optimizer_tracking/index.md: Tracking optimizer performance + - tutorials/streaming/index.md: Streaming responses + - tutorials/async/index.md: Asynchronous processing + - production/index.md: DSPy in production environments + API Reference - Core: + - api/index.md: API documentation overview + - api/modules/*.md: Core modules - Predict, ChainOfThought, ReAct, etc. + - api/signatures/*.md: Signature system - InputField, OutputField, Signature + - api/primitives/*.md: Core primitives - Example, Prediction, History, etc. + API Reference - Optimizers: + - api/optimizers/*.md: All optimizers - BootstrapFewShot, COPRO, MIPROv2, etc. + API Reference - Adapters and Tools: + - api/adapters/*.md: Input/output adapters - ChatAdapter, JSONAdapter, etc. + - api/evaluation/*.md: Evaluation metrics and tools + - api/models/*.md: Language model interfaces + - api/tools/*.md: Built-in tools - ColBERTv2, Embeddings, PythonInterpreter + - api/utils/*.md: Utility functions and helpers + Community: + - community/community-resources.md: Community resources + - community/use-cases.md: Real-world use cases + - community/how-to-contribute.md: Contributing guidelines extra: social: diff --git a/docs/requirements.txt b/docs/requirements.txt index 8083e88fe6..36badd2323 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -5,5 +5,6 @@ mkdocs-material[imaging] mkdocs-redirects mkdocstrings mkdocstrings-python +mkdocs-llmstxt>=0.3.0 urllib3==1.26.6 mistune==3.0.2