The rapid adoption of generative AI is transforming industries worldwide. According to McKinsey's 2024 report, AI adoption has surged to 72% of organizations, with 65% of respondents reporting regular use of generative AI in at least one business function.This rapid growth emphasizes the need for cost-efficient, customizable, and privacy-focused solutions like small-size open-source large language models (LLMs). However, as highlighted in our <a href=https://recursiveai.co.jp/en/blog/best-open-source-llm-models/>benchmark test results</a>, these models often fall short in performance compared to their larger counterparts. To address this, we explored fine-tuning and prompt optimization as strategies to close the gap.<h1>What Are Small Open-Source LLMs?</h1>Small open-source LLMs are publicly accessible AI models with fewer parameters (under 10B), making them deployable on standard hardware such as consumer-grade PCs. Their accessibility makes them an attractive option for organizations and individuals with limited technical resources.<h3>Key Advantages</h3><ul><li>Cost Efficiency: Operate on standard systems with minimal resource requirements, reducing costs.</li><li>Customizability: Can be fine-tuned to specific tasks, delivering tailored solutions.</li><li>Privacy Assurance: Deployed locally, eliminating the need for external APIs and ensuring data security.</li></ul>In this article, we focus on optimizing three popular small open-source models: Qwen 2.5-7B (Alibaba), Llama 3.1-8B, and Llama 3.2-3B (Meta).<h1>Evaluating RAG Capabilities of Small Models</h1>We assessed the models using Recursive’s proprietary <a href=https://recursiveai.co.jp/en/blog/introducing-flow-benchmark-tools/>Flow Benchmark Tools</a>, designed to evaluate models on real-world tasks. The benchmark utilized a dataset of Japanese government documents paired with challenging questions to test two critical capabilities:<ol type=1><li>Question-Answering Using Retrieval-Augmented Generation (RAG)</li><li>Whole Document Analysis</li></ol>Flow Benchmark Tools evaluated model performance across English and Japanese, using a scoring system from 0 (worst) to 10 (perfect).<h1>Prompt Optimization: A Low-Cost Enhancement</h1>Prompt optimization involves refining how instructions or input text are presented to an AI model. Think of it as crafting a clear, well-structured directive to guide the model’s response. This low-cost strategy is often the first step to improving performance without altering the model itself. In our experiments, the optimizations were focused on document analysis, as question-answering with RAG relies more heavily on retrieval quality, which is less influenced by prompt adjustments.<h3>1. Query Placement Improves Accuracy</h3>The position of the query within the prompt significantly influenced results. For example, placing the query after the document block for the Qwen 2.5-7B model improved its score from 6.35 to 6.88. This demonstrates the value of careful prompt design in enhancing performance.<img src=/content/images/image_a9c925f5c6ecdfd963de592a5af2e4b7.png><h3>2. Addressing Hallucinations</h3>Hallucinations occur when AI models generate incorrect or fabricated answers with confidence. For example, a model might invent URLs that do not exist or provide information absent from the input data. This issue is particularly problematic for tasks requiring factual accuracy, such as document analysis.In our tests with Llama 3.1-8B, minor prompt adjustments were insufficient to mitigate hallucinations. However, using a structured data prompt—adapted from DSPy—significantly reduced hallucinations, improving the model’s performance score from 4 to 8. This template defines clear input fields and enforces strict output formats, ensuring the model produces accurate and consistent responses.Structured data prompt template (adapted from DSPy):<img src=/content/images/image_609d34255c09d42c7bb9d717a8939030.png><h1>Fine-Tuning LLMs on a Custom Dataset</h1>Fine-tuning adapts pre-trained AI models by further training them on a specialized dataset aligned with specific tasks. Using LoRA (Low-Rank Adaptation), we efficiently trained the models by adding small, trainable modules to existing layers, which adjust weights without changing the original parameters, reducing memory usage and speeding up training. To align the models with our benchmark tasks, we created a dataset modeled on the Flow Benchmark instruction style. Fine-tuning was applied to two models, Llama 3.1-8B and Llama 3.2-3B, and produced mixed results depending on the task.<h3>1. Question-Answering with RAG</h3><ul><li>Llama 3.1-8B showed negligible improvement.</li><li>Llama 3.2-3B achieved a 7% improvement.</li></ul>The limited improvement stems from the task’s reliance on information retrieval—a capability the models already handled well. Fine-tuning focused more on helping the models follow task-specific instructions than introducing new knowledge. Since the Llama 3.1-8B model’s performance was already comparable to larger models like Llama 3.1-70B, further fine-tuning provided minimal additional benefits.<img src=/content/images/image_0f27e302009957b243bf35870cabec53.png><h3>2. Whole Document Analysis</h3><ul><li>Llama 3.1-8B improved by 10%.</li><li>Llama 3.2-3B achieved a 26% boost.</li></ul>These significant improvements resulted from the fine-tuning process, which enhanced the models’ ability to understand and adapt to document analysis tasks, leading to higher scores.<img src=/content/images/image_5a5d38fca4f4bfeec3a22ea3be4a0a26.png><h1>Key Takeaways</h1>The results presented in this article highlight the importance of fine-tuning and prompt optimization in maximizing the potential of small open-source LLMs:<ol type=1><li>Prompt Optimization: While thoughtful prompt design can improve model performance in many cases, addressing hallucinations remains a complex challenge. Various approaches exist, but most lack general applicability. However, in our experiments, the structured data prompt proved highly effective. While this approach worked in our scenario, it should be seen as one of many potential methods rather than a universal solution.</li><li>Fine-Tuning: Aligning fine-tuning datasets with task-specific requirements is crucial. Although fine-tuning showed limited impact on question-answering with RAG, it delivered substantial improvements for complex tasks like document analysis, enabling more accurate and reliable outputs.</li></ol>By combining these techniques, small open-source models can narrow the performance gap with larger alternatives while maintaining advantages in cost, customization, and privacy.At Recursive, our commitment to open-source technologies reflects our vision of democratizing AI while empowering enterprises to build fairer, more sustainable solutions. Reach out to us at sbdm@recursiveai.co.jp to discuss how our tools can enhance your AI strategy.

How can small open-source LLMs rival larger models? Discover how Recursive used fine-tuning and prompt optimization to reduce hallucinations and achieve a 26% improvement in document analysis performance. 

Fine-Tuning Small LLMs: Improved Performance with Fewer Resources

At Recursive, we are committed to delivering cutting-edge AI solutions, and today, we are excited to share the results of our extensive evaluation of open-source LLM models. As more businesses embrace open-source technologies for their transparency and customizability, one key question remains: Can these models match the performance of closed-source alternatives like GPT-4?To address this, we used our recently launched <a href=https://recursiveai.co.jp/en/blog/introducing-flow-benchmark-tools/>Flow Benchmark Tools</a> to thoroughly test the performance of leading open-source models by integrating them into FindFlow, a platform developed by Recursive for text generation, search, and analysis.The tests focused on assessing two critical functions: question answering using retrieval-augmented generation (RAG) and whole document understanding. These capabilities form the core of our FindFlow platform as SearchAI and AnalysisAI features.<h1>What Are Open-Source LLM Models?</h1>Open-source LLMs are publicly accessible models whose code, architecture, and sometimes training data are made available for use, modification, and redistribution. Businesses can leverage these models for their own applications, taking advantage of three key benefits:<ul><li>Customizability: Models can be fine-tuned to meet specific business needs.</li><li>Transparency: Full visibility into the model’s architecture and inner workings.</li><li>Cost Efficiency: Open-source models eliminate the need for subscription fees or API costs associated with closed-source systems.</li></ul><h2>Open-Source vs. Closed-Source LLM Models: A Quick Comparison</h2><img src=/content/images/image_fe558b3edda755ded978b0b04f54b96c.png><h1>Flow Benchmark Tools: A Closer Look</h1>To provide accurate and actionable insights, we used Recursive’s proprietary Flow Benchmark Tools to test SearchAI and AnalysisAI features of FindFlow on several open-source LLM models provided by top companies such as Google, Microsoft, Meta, and others.<ul><li>SearchAI focuses on question-answering using Retrieval-Augmented Generation (RAG), assessing how well models retrieve accurate, context-specific information from documents.</li><li>AnalysisAI centers on whole document understanding, evaluating how effectively models extract insights from complex, full-length documents.</li></ul>A unique feature of our tools is the multilingual capability, allowing us to evaluate model performance not only in English but also in other languages, with a specific focus on Japanese. This multi-language assessment makes our benchmarks particularly relevant for global applications.The approach used in the Flow Benchmark Tools mirrors real-world use cases by utilizing a dataset of Japanese government documents with challenging questions to ensure a thorough evaluation. It employs a multi-model approach, including state-of-the-art models such as GPT-4, Claude 3, and Gemini, to generate objective results, with ratings from 0 (worst) to 10 (perfect). For more details, check out our <a href=https://recursiveai.co.jp/en/blog/introducing-flow-benchmark-tools/>launch announcement article</a>.Our Flow Benchmark Tools are publicly available on<a href=https://github.com/recursiveai/flow_benchmark_tools> GitHub</a> and<a href=https://pypi.org/project/flow-benchmark-tools/> PyPi</a>.<h1>Performance Overview: Evaluating Open-Source LLM Models by Size</h1>It’s essential to analyze open-source LLM models by size, as performance varies significantly between big, medium, and small models. Larger models generally deliver superior accuracy, especially in complex tasks, while smaller models offer a cost-efficient solution, capable of running on personal computers with lower hardware requirements.To establish a performance benchmark, we included GPT-4o, a closed-source model renowned for its strong capabilities. When integrated with FindFlow, it achieved impressive scores of 8.27 for SearchAI and 8.9 for AnalysisAI.<h2>Big Size Models: Key Insights</h2><img src=/content/images/image_7c3d51a86cc5c0025effb7b5b65332c5.png><ul><li>Llama 3.1-70B (Meta) and Qwen 2.5-32B (Alibaba) emerged as top performers among the larger models. Both demonstrated robust capabilities, making them ideal for more complex tasks.</li><li>Llama 3.1-70B showed particularly strong instruction-following in English but lacked built-in support for Japanese, which could be improved through fine-tuning.</li><li>Mistral small-22B (Mistral AI) delivered lower scores in Analysis AI, especially for Japanese tasks, limiting its effectiveness in complex document understanding.</li><li>Gemma 2-27B (Google) showed weaker overall performance, particularly when handling larger documents. By design, it supports an 8,000-token context length, which is significantly lower than other models. However, its effectiveness drops when the context exceeds 7,000 tokens, limiting its capability for complex tasks.</li></ul><h2>Medium Size Models: Key Insights</h2><img src=/content/images/image_6aa4b8d39362eeccd6006f509f413c04.png><ul><li>Qwen 2.5-14B (Alibaba) led the medium-size category, delivering strong results in both English and Japanese queries, with particularly high AnalysisAI scores.</li><li>Mistral Nemo-12B (Mistral AI) showed strong English SearchAI scores but lagged in Japanese SearchAI, indicating inconsistencies across languages.</li><li>Phi-3-medium-14B (Microsoft) underperformed in AnalysisAI compared to other mid-size models, highlighting limitations in handling complex document understanding tasks. Additionally, Japanese is not on the list of supported languages, restricting its usefulness for multilingual analysis.</li></ul><h2>Small Size Models: Key Insights</h2><img src=/content/images/image_767fd7ce013d03534b06b81ce26e9bd4.png><ul><li>GLM4-9B (Zhipu AI) demonstrated top-tier performance, surpassing many larger models in SearchAI and AnalysisAI tasks.</li><li>Llama 3.1-8B (Meta) showed solid performance in SearchAI. In AnalysisAI, it performed moderately well for its size, but its Japanese capabilities lagged behind. Fine-tuning could enhance its ability to process Japanese queries more effectively.</li><li>Qwen 2.5-7B (Alibaba) showed respectable performance relative to its size, excelling in certain tasks, making it a competitive model in the small-size category.</li><li>Gemma 2-9B (Google) underperformed compared to other small models. Although it supports an 8,000-token context length, its effectiveness significantly declines beyond 5,000 tokens, posing challenges for handling more demanding tasks or long-form content.</li></ul><h1>Open-Source LLMs Show Promise with the Right Integration</h1>Our Flow Benchmark results showcased the top-performing open-source LLM models across three size categories. For big-size models, Llama 3.1-70B (Meta) and Qwen 2.5-32B (Alibaba) excelled in both SearchAI and AnalysisAI. In the medium category, Qwen 2.5-14B (Alibaba) stood out, particularly for its multilingual strengths, performing well in both English and Japanese. Among small models, GLM4-9B (Zhipu AI) impressed by outperforming larger models in accuracy and efficiency.These findings show that with the right integration—like our FindFlow platform—open-source LLMs can achieve performance on par with closed-source models while providing greater flexibility and cost savings. To discover how FindFlow can save your business 20+ hours a month and guide you in selecting the best platform, reach out to us at <a href=mailto:sbdm@recursiveai.co.jp>sbdm@recursiveai.co.jp</a>.

This article reveals the results of testing top open-source LLM models to see if they can match closed-source alternatives like GPT-4. Using Recursive’s Flow Benchmark Tools, we evaluated models from Google, Microsoft, Meta, and others on question-answering with Retrieval-Augmented Generation (RAG) and whole document understanding.

Best Open-Source LLM Models: Flow Benchmark Results Revealed

In late February and early March of 2024, a team from Recursive embarked on a significant business trip to Indonesia. The primary objective was to enhance their AI model designed to predict future Ground Water Levels (GWL) in the tropical peatlands of West Kalimantan. This initiative aims to prevent forest fires, anticipate floods, and ensure the growth of plants. The team, consisting of <a href=https://www.linkedin.com/in/qsiutkowski/>Quentin Siutkowski</a>, <a href=https://www.linkedin.com/in/dmitrylyamzin/>Dmitry Lyamzin</a>, and <a href=https://www.linkedin.com/in/david-landup-859455144/>David Landup</a>, had been working on the GWL project for over two years and faced numerous challenges that required on-site observations and experiments to resolve. Discover their insights and passion from this through an interview!<img src=/content/images/image_c82ef93b5dfc1704c2ecc0f8e3235c71.png>From left to right: Quentin, Dmitry, and David enjoying a meal.　<a href=https://borealis.jp/en/>Borealis</a> is an AI model as one of Recursive’s 4 Technology Platforms, using advanced physics-based modeling and AI to improve environmental management. It helps with reforestation, carbon storage, water resource preservation, and minimizing fires and ecosystem degradation. Borealis integrates large-scale climate and geological data, hybrid environment simulation, satellite data, and ground observations to support sustainability use cases like water level management, fire hazard mapping, renewable energy forecasting, and power demand forecasting. Here the project focuses on groundwater levels. 　Question #1 : How was it to finally go there directly when you've worked on this project for more than two years?Dmitry: It was very exciting! We finally got to see where the data comes from. Riding a speedboat for three hours into the jungle and then arriving to find chickens everywhere was surreal. It was a unique experience that helped us understand their daily challenges and the life they live. Being there allowed us to develop personal connections with them, which brought a lot of motivation.　Quentin: People who work on the site live there, so it's an isolated area with everything needed to sustain themselves. It was different from just seeing images and data, we could play volleyball, eat, and talk with the local team, which was very motivating.　David: There were always be question marks about the things we've done, the assumptions we've made, and not just resolving those questions, but finally seeing what we spent so much time trying to understand in real life was a surreal feeling. I remember sending messages to some of our colleagues who also worked on the project beforehand saying “This is the thing we couldn’t figure out a year ago!”. So finally being able to see those in real life was a pretty surreal feeling.　Question #2 : Did this trip allow you to improve your solution Borealis? How, if yes?Dmitry: We definitely learned a lot about the irrigation system they use there! We will use this knowledge in future model developments. We finished a certain phase and delivered a product before we went, so we haven’t fully used the acquired knowledge just yet, but we will in the future. 　Question #3 : So when you were there, did you receive any feedback on the work you did before?David: We had multiple discussions with people there, both up and down the chain of management, and received good feedback on what we did as well as potential areas for improvement. The dynamics of the area are complex, making it difficult to model and understand what's happening. However, we gained valuable insights into the daily problems people face and what could be resolved by our technology. Overall, it was a gratifying experience.　Quentin: Communication between the head company and the subsidiary isn't straightforward; it's restricted to specific channels and a few key individuals. Engineers on the ground have certain concerns and ideas that people in Tokyo don't necessarily share. We strive to balance the needs of both the head company and the engineers, which is a challenging task but ultimately benefits everyone involved. Our prior interactions with them established a foundation of trust, evident in the warm reception and welcome we received as guests. They recognized our genuine efforts to improve their lives and work collaboratively, which further deepened the mutual trust between us.<img src=/content/images/image_4055e5c54365574ceaaba5f9ba03f479.png>David getting on the speedboat.　Question #4 : How do you feel this project aligns with the values and goals of your team?Dmitry: This project is directly related to our values and goals as a company. It's a sustainable operation in the jungle where people genuinely care about their business and its impact on the wider territory. They care about the wildlife and have dedicated personnel responsible for conservation management, which aligns very well with our values.　Quentin: I think it's the flagship project of the company when it comes to sustainability. While we are getting more projects aligned with our values, this one is the bread and butter of the company. The impact extends far beyond just the facilities (physical infrastructure and operational areas that are part of the project), as most of the area consists of conservation zones. They shared videos of orangutans and other endangered species they are protecting. They are putting cameras and sensors to keep track of the populations and ensure they are not disturbing them.　David: Highly aligned, and has the potential to prevent massive carbon-emitting events. Peat has extremely high carbon sink effects, capturing a lot of carbon that gets stored in the soil and also helps massive trees grow into rainforests. This biomass is then used to build housing, essentially converting CO2 from the air into housing for people. Tropical forests and peatlands are particularly good at this, but they require specific weather conditions, making it crucial to maintain the existing ones. For this specific reason, peat is also very flammable and it’s easy to re-release that same captured carbon back into the atmosphere. We're hoping that our work may mitigate fire hazards, such as the 2015 fire that burned 2.6m hectares of forests in Indonesia, causing up to ~1.3B tonnes of CO2 to re-enter the atmosphere, which is higher than the yearly CO2 emissions of many countries. This was a planet-scale event.<img src=/content/images/image_8ae4cc11afb62508fde919f2138c1925.png>From left to right: Quentin, Dmitry, and David at the airport.　Question #5: How do you see the Borealis team evolving now as a result of this project?Dmitry: Now that we've completed this project and been there, we have gained more expertise. We could look at growing in the future by taking on a bigger area and more long-term contracts.　Quentin: The Borealis team will expand to support projects similar to this one, and we are also working on other projects, including long-term ones with other companies. Sometimes we see connections between projects, even if they seem far apart on paper. Some are based in Japan, some in Indonesia. The environments are different, but the approach, knowledge, and systems we develop can be reused or adapted. We have many projects related to water, which is a common thread that connects them. We also have many exciting ideas for Borealis.　<img src=/content/images/image_2c7087d072bf5366404de2329b8df14a.png>Dmitry in the speedboat.　Question #6: How can Borealis concretely help?Dmitry: It was a good surprise to see how sophisticated their water operations and canal systems are. They know what they're doing and need minimal intervention. Our approach may not be to tell them what to do daily, but to support them in their activities. They are already doing a lot of good things, but there's still a lot of guessing. We hope to make their guessing more accurate and science-based.　Quentin: We can support them in their daily activities, even though they are already doing well. There are factors like changes in evapotranspiration, and rainfall that affect groundwater, which their system may not fully account for. Our solution can help them. It didn't rain much while we were there, but on the last day, it rained heavily. They said all the work done in the past two days had to be undone because of the rain. We hope to help avoid such situations in the future and make their life easier.　Question #7 : How was your experience interacting with the local communities there?Dmitry: People were very friendly and the homemade food was amazing. Quentin played volleyball with them as well. I didn't because it was too hot. It was just too hot to function.　Quentin: People were very kind and welcoming. It was nice to exchange with them and not feel like outsiders. We created something more than just a visitor relationship. The person we had talked to before was an expert of the area and respected by the people there. It was good to see him in his natural environment and get his support. 　David: The atmosphere was filled with smiles, and people seemed genuinely happy. They were incredibly kind and welcoming towards the work we were doing. Although there was a bit of a language barrier since they didn't speak English and we didn't speak Bahasa, we managed to communicate effectively using our hands. It was a wonderful experience, I wish we could have stayed a bit longer to enjoy more of the beauty and the fun environment.<img src=/content/images/image_34a7fe6472c3a64ff473a59c91a327e0.png>David and Dmitry getting to do some ground water level measurements.　Question #8: What is the next step?Quentin: We are talking about expanding the project to newer areas and seeing what's possible. It's not decided yet, but the project has huge potential. We are optimistic that we'll do even more and have an even bigger impact.　David: The next step would be to generalize the solution to other areas. There are massive zones this could be used for, both for planning where to create irrigation canals and for long-term analysis and future forecasts. <img src=/content/images/image_fa85e249a52911778b63dd23af3ca12d.png>Quentin and David at a briefing meeting before going to the field.　Recursive is a service provider of AI solutions for building a sustainable future. We provide AI system development and consultation services by combining our knowledge of diverse industries such as environment, energy, medical, pharmaceutical, food, and retail with our advanced technological capabilities and expertise in sustainability business. Our unparalleled professionals lead the way in creating a new society with world-class technology in order to leave a better global environment and society for future generations.　　Please visit our Borealis web page here:
<a href=https://borealis.jp/en/>https://borealis.jp/en/</a>　Our 4 technology platforms are following; <img src=/content/images/image_3d2d7bdae0543e3d6d1fa9cb1d4352b0.png>

In late February and early March of 2024, a team from Recursive embarked on a significant business trip to Indonesia. The primary objective was to enhance their AI model designed to predict future Ground Water Levels (GWL) in the tropical peatlands of West Kalimantan. This initiative aims to prevent forest fires, anticipate floods, and ensure the growth of plants. 

[Borealis Project background report] Indonesia Business Trip : February 27 - March 2, 2024

Today, we're proud to present the Flow Benchmark Tools, designed to complement and strengthen our FindFlow platform. This open-source release marks a significant milestone in our commitment to advancing retrieval-augmented generation (RAG) technology and empowering the global AI community.The Flow Benchmark Tools represent a major leap forward in our ability to evaluate and optimize RAG systems. As an extension of our FindFlow suite of platforms, these tools address a critical need in the AI landscape: providing a standardized, comprehensive method for benchmarking RAG pipelines.Our goals for the Flow Benchmark Tools are ambitious yet focused:<ol type=1><li>To establish a new standard for assessing RAG system performance, particularly in multilingual contexts.</li><li>To empower developers and researchers with the tools they need to push the boundaries of RAG technology.</li><li>To foster transparency and collaboration within the AI community by open-sourcing our benchmarking methodology.</li></ol><h3>Understanding RAG and LLMs: Bridging the Knowledge Gap</h3>Before diving into the specifics of our benchmarking tools, it's crucial to understand the technologies they're designed to evaluate. Large Language Models (LLMs) are the backbone of modern AI systems, leveraging vast neural networks to understand and generate human-like text. These models, measured by the number of parameters they contain, excel at capturing the general patterns of language use.However, LLMs have limitations when it comes to accessing current or highly specific information. This is where Retrieval-Augmented Generation (RAG) comes into play. RAG is a powerful technique that enhances the capabilities of LLMs by incorporating external knowledge sources. It allows AI systems to combine the fluency and reasoning capabilities of LLMs with up-to-date, factual information retrieved from external databases.The importance of RAG cannot be overstated. It enables a wide range of use cases where accuracy and reliability are paramount. In customer support, RAG ensures that AI assistants provide accurate, up-to-date information about products, services, and policies, leading to improved customer satisfaction and reduced workload for human agents. The education sector benefits greatly from RAG, as it allows educational AI tools to access and present the most current knowledge in various fields of study, keeping learners at the forefront of their disciplines. In the realm of industrial and academic research, RAG empowers researchers with AI assistants that can draw upon vast libraries of scientific literature and data, accelerating the pace of discovery and innovation. Content creators also find immense value in RAG systems, which help them by providing accurate, factual information to support their work, enhancing the quality and credibility of their output.By combining the strengths of LLMs with external knowledge retrieval, RAG significantly reduces the risk of AI hallucinations – instances where models generate plausible but incorrect information. This makes RAG-enhanced systems more trustworthy and applicable in critical domains where factual accuracy is essential.<h3>State-of-the-art benchmarking capabilities</h3>The Flow Benchmark Tools offer unparalleled insights into RAG system performance. By addressing the complexities inherent in RAG pipelines—including semantic retrieval, query generation, and LLM-based answer generation—our tools provide a nuanced, comprehensive evaluation framework.While there are numerous LLM benchmarks available in the field, such as LMSYS's Chatbot Arena and Arena-Hard, SEAL Leaderboards, and ChatRAG-Bench, the Flow Benchmark Tools stand out for their unique approach. Unlike most benchmarks that focus on general queries or pre-processed documents, our tools emphasize document-specific information retrieval and end-to-end document processing. This approach more closely mirrors real-world scenarios, making our benchmarks more product and end-user focused.By evaluating the entire pipeline from raw document processing to information retrieval and response generation, the Flow Benchmark Tools provide a more holistic and practical assessment of RAG system performance. This comprehensive approach aligns with our goal of developing RAG technologies that excel not just in controlled environments, but in the complex, document-rich scenarios that businesses and researchers encounter daily.We're especially proud to announce that our initial release focuses on Japanese language performance, setting a new benchmark for non-English RAG systems. This emphasis on multilingual capability reflects our commitment to developing AI technologies that serve a global audience.In conjunction with this release, we're sharing benchmark numbers that compare FindFlow's performance against several leading RAG systems. These results demonstrate the power and versatility of our approach, while also providing valuable data to the wider AI community.<h3>Embracing the open-source ethos</h3>By open-sourcing the Flow Benchmark Tools, we're inviting the global AI community to join us in advancing RAG technology. We believe that collaboration and transparency are key to unlocking the full potential of AI, and we're excited to see how developers, researchers, and companies will leverage and build upon our work.Our release of the Flow Benchmark Tools is just the beginning. We're committed to continually improving and expanding these tools, with plans to incorporate additional languages, extend our benchmarking capabilities, and refine our methodologies based on community feedback.We can't wait to see how the AI community will use the Flow Benchmark Tools to drive innovation, improve RAG systems, and ultimately create more powerful, accurate, and reliable AI applications. We look forward to your feedback and contributions as we work together to shape the future of AI technology.Our Flow Benchmark Tools are now <a href=https://github.com/recursiveai/flow_benchmark_tools>publicly available on Github</a> and <a href=https://pypi.org/project/flow-benchmark-tools/>PyPi</a>.<h3>FindFlow demonstrates state of the art performance</h3>We're excited to share the results of our comprehensive benchmarking efforts, which highlight FindFlow's exceptional performance in two critical areas: Question Answering using RAG (Retrieval-Augmented Generation) and Whole Document Understanding. These benchmarks not only showcase the capabilities of our SearchAI and AnalysisAI pipelines but also demonstrate how FindFlow stands out in the competitive landscape of AI-powered document analysis tools.<h3>Benchmark methodology</h3>Our benchmarking process was designed to be thorough, fair, and reflective of real-world scenarios. We used a dataset of Japanese government documents paired with challenging questions, which we plan to open-source soon to contribute to the wider AI research community. To ensure the most objective and robust evaluation possible, we employed an automated evaluation system that leverages multiple state-of-the-art LLMs, including GPT-4, Claude 3, and Gemini. This multi-model approach allows us to mitigate potential biases and provide a comprehensive assessment of performance. The evaluation system outputs a mean opinion rating with values ranging from 0 (worst) to 10 (perfect).<h3>RAG question answering: SearchAI shines</h3>In the realm of RAG question answering, our FindFlow SearchAI pipeline was put to the test against two popular question answering pipelines: OpenAI Assistant and the default LangChain RAG pipeline. The results speak for themselves:<ul><li>FindFlow SearchAI achieved a mean rating of 8.42</li><li>OpenAI Assistant followed with a rating of 7.81</li><li>LangChain's default RAG pipeline scored 7.01</li></ul>These scores clearly demonstrate FindFlow SearchAI's superior performance, outpacing the nearest competitor by a significant margin of 0.61 points and the third-place solution by 1.41 points.<img src=/content/images/image_5e5b06ca4e88b94f153d3b382c3f9b9b.png><h3>Document analysis: AnalysisAI leads the pack</h3>For whole document understanding, we benchmarked our FindFlow AnalysisAI against OpenAI Assistant and Gemini's long-context LLM (Gemini-1.5-Pro). The results once again highlight FindFlow's capabilities:<ul><li>FindFlow AnalysisAI achieved an impressive mean rating of 8.90</li><li>Gemini-1.5-Pro secured the second position with a rating of 7.22</li><li>OpenAI Assistant followed with a score of 6.29</li></ul>In this category, FindFlow AnalysisAI's performance surpasses the runner-up by 1.68 points and the third-place contender by a substantial 2.61 points.<img src=/content/images/image_3a450bab77fc62571659cd54b677701c.png>As we continue to refine and enhance our technology, we remain dedicated to maintaining this high standard of performance and to providing our users with the most advanced, reliable, and effective document analysis tools available in the market.

Today, we're proud to present the Flow Benchmark Tools, designed to complement and strengthen our FindFlow platform. This open-source release marks a significant milestone in our commitment to advancing retrieval-augmented generation (RAG) technology and empowering the global AI community.

Introducing Flow Benchmark Tools

In the realm of business and operations, optimizing complex systems is a central challenge, particularly in sectors like manufacturing, pharmaceuticals, and power production. These industries often deal with extensive operational pipelines where the performance of one component can have cascading effects on others, creating a network of interdependencies. Traditionally, the optimization of these systems is handled one step at a time, which may not capture potential efficiencies or the broader impact of changes across the system.At Recursive, we develop methods that optimize such systems from end-to-end. This approach increases production efficiencies and reduces greenhouse gas emissions by enhancing the overall system's performance rather than its individual parts. Our approach surpasses traditional optimization methods that struggle with the complexity and scale of these tasks and are unsuitable for achieving the desired outcomes efficiently.<img src=/content/images/image_1e16d1c103f8f212a190e4540bb49970.jpg>Zenith, our deep learning platform, addresses these challenges by leveraging computational optimization techniques that significantly improve upon the solutions that human experts can create. By adopting AI-driven strategies, Zenith is capable of exploring a wider range of potential solutions quickly and with greater precision, thus enabling more effective system-wide optimizations.<h2>Why is Combinatorial Optimization Difficult?</h2>Traditional methods struggle significantly in the realm of combinatorial optimization due to the nature of the problems at hand. Combinatorial optimization involves manipulating discrete parameters, akin to repositioning pieces on a chessboard to achieve the best strategic setup. Each piece can be placed on specific squares, and each potential configuration could vastly differ from another, impacting the outcome significantly.This contrasts sharply with traditional optimization tasks, such as determining the optimal speed for a car to maximize fuel efficiency. In such scenarios, even minor adjustments to speed result in small, predictable changes in fuel consumption, allowing for a smooth and gradual optimization process. This is visually represented by a smooth landscape where gradual adjustments lead towards an optimal point (as shown in the image below).<img src=/content/images/image_0c7a28b5d15f0d5a2fffb043b77e7e80.png>In contrast, combinatorial optimization does not allow for such minor tweaks. Since the solutions involve discrete choices (like switches being either on or off), there is no way to 'nudge' the solution slightly in one direction and measure the effect. This absence of a gradient to follow makes the problem non-continuous and dramatically increases the complexity of finding the optimal solution. Consequently, most approaches resort to trial and error or rely on simple heuristic methods, which are computationally intensive and often fail to locate the best solutions. This methodological gap underscores the need for innovative approaches like those implemented by Zenith, which rethink how these challenging problems are tackled.<h2>Examples Where Combinatorial Optimization Can Help</h2>The potential of combinatorial optimization to revolutionize diverse sectors is immense, with each application bringing specific benefits that translate into direct business advantages.<h3>Delivery Route Optimization and Logistics</h3>In the logistics industry, optimizing delivery routes is crucial for efficiency. Currently, routes are often planned based on simple heuristics that consider distance but might ignore other variables like traffic patterns or delivery time windows. This can lead to suboptimal paths that waste time and fuel. By applying combinatorial optimization, delivery routes can be strategically designed to account for multiple factors simultaneously, such as shortest paths, least congested routes, and specific time slots for deliveries. This optimization not only reduces fuel consumption and delivery times but also enhances customer satisfaction through more reliable delivery schedules. A logistics company that does not implement best-in class route optimization not only incurs wasted costs, it risks losing market position to competitors that will outperform them using these technologies.<h3>Drug Formulation Optimization in Pharmaceuticals</h3>The process of drug formulation involves finding the optimal combination and proportion of various ingredients to achieve a desired therapeutic effect while minimizing side effects. Traditionally, this process is performed by trial and error, consuming considerable time and resources. Combinatorial optimization transforms the approach by treating the formulation as a complex puzzle where each combination of ingredients and their quantities can be precisely calculated to optimize efficacy. This method can significantly speed up the development process, reduce costs, and bring more effective drugs to market faster, providing a competitive edge to pharmaceutical companies.<h3>Industrial Machinery Optimization</h3>In the manufacturing sector and related fields,, the arrangement of machinery and the settings of complex systems like HVAC are critical for operational efficiency. Traditionally, the layout of factory floors and the settings of HVAC systems are often determined based on static criteria that do not account for dynamic operational conditions. Combinatorial optimization enables a more dynamic approach by considering various operational scenarios to determine the most efficient layouts and settings. For example, optimizing the placement of machinery can minimize the time materials are in transit between processes, reducing cycle times and increasing productivity. Similarly, optimizing HVAC settings in real-time based on factors such as occupancy and external weather conditions can drastically reduce energy consumption. The result is a significant reduction in operational costs and an enhancement in production efficiency, contributing directly to the bottom line.Businesses that are leveraging combinatorial optimization are achieving a level of operational efficiency and product effectiveness that is not attainable with traditional methods. This optimization not only brings cost savings but also fosters innovation, setting the stage for next-generation infrastructure and operations.<h2>How Zenith Addresses Combinatorial Optimization Challenges with AI</h2>Zenith's approach to combinatorial optimization incorporates advanced techniques from machine learning and graph theory to tackle the complex nature of these problems efficiently. The following is a detailed breakdown of the processes involved:<h3>Graph Representation</h3>Zenith models the elements of the problem as nodes in a graph, with the relationships between these elements represented as edges. In drug formulation, each ingredient would be a node and the chemical interactions between different ingredients would be the edges. This graph-based approach allows for a structured analysis of the problem, providing a clear framework to assess how changes in one element affect others.<img src=/content/images/image_7b558609b5dab79b402aa8a6c694b6bf.png><h3>Embeddings for Smoothing</h3>Due to the discrete nature of combinatorial optimization problems, traditional gradient-based optimization methods are not applicable. Zenith utilizes embeddings to map these discrete elements into a continuous, high-dimensional space. This mapping process transforms the optimization landscape into a form that is more amenable to computational analysis and manipulation. The embeddings help to identify potential directions for exploration in the solution space, creating a way to estimate the impact of altering different elements even though the problem itself does not allow for small, continuous adjustments.<img src=/content/images/image_a4e20480963795c48b3deee7006866cb.png><h3>Discrete Optimizers</h3>Zenith employs specialized optimization algorithms designed to handle the discrete and structured nature of the problems. These include evolutionary algorithms and graph neural network optimizers. Evolutionary algorithms are particularly effective as they generate a wide range of potential solutions and evolve these solutions over time based on their performance, mimicking natural selection. Graph neural network optimizers leverage the graph structure of the problem, enabling the optimizer to consider the complex interactions and dependencies between elements to suggest optimal configurations.This methodological framework ensures that Zenith can systematically explore and evaluate a vast array of possible solutions, significantly increasing the probability of identifying the most effective configurations. This process not only streamlines the search for optimal solutions but also reduces the computational load compared to traditional methods, which often rely on exhaustive search or simplistic heuristics.Zenith’s technology strength goes beyond traditional methods by combining discrete optimization algorithms with deep learning features, harvesting the best of two worlds. Discrete optimization algorithms by themselves are often limited to perform optimization of problems that can be described in explicit mathematical detail. Deep learning methods on the other hand find data-driven solutions, which means that the problem description is inferred from examples. Such data-driven solutions require a lot of examples and deep learning method can often only draw limited benefits from explicit problem formulations, even if they are available. The patterns developed in Zenith allow a combination of both approaches, resulting in a system that can perform optimization for extremely complex problems, taking full advantage of available data and available expert knowledge, providing interpretable solutions following controllable parameters.<img src=/content/images/image_e48ce0df3c07e24db4889d647353cfa7.png><h2>Use Cases for Zenith</h2>Zenith’s use of these advanced technologies aims to transform physical infrastructure into highly optimized systems. We can significantly boost productivity in industries such as manufacturing, energy, and transportation, by increasing the efficiency of layouts and operational sequences. A more efficiently designed factory floor layout, optimized for minimal movement of materials, can reduce production times and lower the energy required for manufacturing operations.By optimizing energy use in large systems, such as HVAC systems in commercial buildings or routing in logistics, we can achieve substantial reductions in carbon emissions. Optimized routes mean fewer miles driven, less fuel consumed, and reduced emissions. Similarly, smartly regulated energy systems in buildings can maintain comfort while minimizing energy waste, contributing to a significant decrease in greenhouse gas emissions.In the pharmaceutical and healthcare sectors, Zenith's optimization technology brings best-in-class solutions to a company’s drug development processes and patient care. Advanced optimization techniques for drug formulation can reduce overall development time and associated costs, and the precise optimization of compound combinations and dosages can lead to more effective drugs with reduced side effects, directly improving patient outcomes.Furthermore, in healthcare management, applying combinatorial optimization to scheduling, appointment systems, and resource distribution can enhance operational efficiency. By optimizing these aspects, healthcare facilities can manage patient flow more effectively, utilize staff more efficiently, and minimize the unnecessary use of resources. These changes can lead to reduced operational costs and improved patient care, providing tangible benefits to healthcare systems.<h2>Future Vision for Zenith</h2>We see a promising potential in the integration of quantum computing within our optimization processes. Quantum computing holds the promise of exponentially faster computations compared to classical computers, particularly suited for problems where potential solutions grow combinatorially. Quantum algorithms could dramatically accelerate the identification of optimal configurations in logistics or complex manufacturing processes, where traditional methods falter due to computational limits.Zenith’s current trajectory includes cutting-edge developments across several domains. We are expanding our toolkit to include large transformer models and state-of-the-art reinforcement learning techniques, which offer significant improvements in handling complex optimization tasks. These AI models can analyze vast amounts of data and learn optimal strategies across a range of scenarios, providing solutions that are both innovative and effective.

An introduction to Zenith, our technological platform to enable increased productivity and accelerated innovation

Introducing Zenith: A Deep Learning Platform for Optimization

Authors: Oktay Kurtulus (Head of Corporate), Rie Le Maître (Corporate Associate)Our recent company retreat was more than just a day away from the office. Taking a moment to step back, reflect, and bond with your team is not only beneficial but also essential to further accelerate our business. That is precisely what we aimed for in our recent one-day company retreat, on Friday, March 15, 2024 and oh, what a day it was! If you are interested in Recursive, please read until the end!<h3>Setting the Stage with Keynote Speeches</h3>Our day kicked off with keynote speeches and objective key results presentations by our leadership team which aimed to provide clarity on our company's overarching vision and objectives for the fiscal year. We used this opportunity to recap our success and failure stories in the previous fiscal year and shared our learnings with the whole team. <img src=/content/images/image_8ff2c00567065373d4550b31faace872.jpg>Our CEO, Tiago, is presenting company vision and strategy for the new fiscal year.<img src=/content/images/image_93e3b607268e1dad1711280835b36064.jpg>Our COO, Toshi, is dropping exciting news regarding company goals.The presentations served two purposes: One, they allowed us to realign our individual goals with the broader organizational mission, fostering a sense of purpose and unity, and two, they helped our new members to get a good grip on our company vision and priorities. The session ended with a lively Q&A session in which many members asked various questions, fostering an insightful, engaging, and an interactive dialogue between the leadership team and members.<img src=/content/images/image_7574b51cb4207141bf75875d9f62410e.jpg>Our members asked many questions to the leadership team following the OKR presentations.<h3>Lunch and Group Photo Session</h3>Before munching on some nice lunch following the presentations, our members took some time to take some memorable group photos in a great setting. As a fast-growing, hybrid-work style company with members spanning Japan and overseas, it was certainly a moment to remember to cherish and capture everyone together.<img src=/content/images/image_48db3e647fe64b0103cd7786ce337265.jpg>Recursive core members, doing their best to make Recursive logo with their hands It is also worth mentioning that for this year's company retreat, we had the exciting opportunity to immerse ourselves in "Link Forest," the brand new training facilities offered by KDDI Learning Inc. KDDI Learning, a distinguished subsidiary of our esteemed client, KDDI, opened their doors for our company retreat and provided us an experience that is nothing short of spectacular, blending state-of-the-art amenities with an innovative learning environment.<img src=/content/images/image_00d4a2e61598af3500df56aa84cd958a.png>Link Forest Training Facilities in Tama, Tokyo. Source: <a href=https://link-forest.jp/>https://link-forest.jp/</a><h3>Icebreaker Workshops: Bonding Through Games</h3>After lunch, we moved on to icebreaker workshops. The aim of these activities was to connect with other team members through games. It was a moment to share the laughter, the mistakes, and the unstoppable sense of unity.<img src=/content/images/image_6bb37ab331904fbc39478cb170a76597.jpg>Project Management team’s out of the box solution to “Tall Tower Challenge.” <img src=/content/images/image_5429ea4d16f08339ccb67afb31e3ea61.jpg>Engineer team’s seems to master the game… do they ?<h3>Workshops Galore</h3>After the icebreaker, we did workshops which were focused on team building and goal setting. These weren't your run-of-the-mill sessions; each workshop was designed to be as engaging as well as educational. While acknowledging our successes is vital, what is equally as important is recognizing areas where we can grow. Through a Strength-Weakness-Opportunities-Threats analysis, we identified key areas for improvement and brainstormed strategies to overcome challenges collectively.<img src=/content/images/image_78ac959265d0800bf2963b471b1fc781.jpg><img src=/content/images/image_fe935cb84ee5dbc7b756ff4c53805d5e.jpg>Our workshops for this retreat mark the initial step towards a comprehensive understanding of our strategic landscape, emphasizing the importance of applying these insights in real-world scenarios. Adhering to the 70-20-10 rule for learning and development, we recognize that the bulk of our valuable knowledge will be acquired through hands-on experiences and collaborative efforts, beyond the confines of such training.<h3>Finishing the retreat with a wholesome Award Ceremony</h3>We also took the time to celebrate our peers with an award ceremony, recognizing their embodiment of our Company Values: Trust & Cooperation, Leadership, Impact, and Challenger Mentality. Among them, one of our youngest members, Shuying, received the Excellence Award for embodying all of our 4 values since joining as a new graduate. From heartfelt speeches to peer recognition, we celebrated our collective achievements. Cheers to our incredible team!<img src=/content/images/image_4ecf081c5ff5c1bee1985c9af7c92351.jpg>For Challenger Mentality, we got 3 recipients! Left to right: David (Machine Learning Engineer), Giuliano (Software Engineer), our CEO Tiago, and Shuying (Business Development).<img src=/content/images/image_9f6c5e91d8983c2884b96e0fb37c4a82.jpg>Our CEO, Tiago (left), awarding Quentin (Project Manager) for both Impact and Trust Cooperation.<h3>From Izakaya to Karaoke</h3>After a day of intense concentration, the evening was our time to let loose. Our first stop? An izakaya in Shinjuku, where laughter and stories flowed. But the fun didn't stop there. Next up was karaoke, a universal equalizer where we cheered each other on regardless of vocal talent, proving that a team that sings together, stays together. And yes, we did discover some hidden talents that night!<img src=/content/images/image_c2c645e523f4fbaf86560ec0c77a2694.png>Our team is nailing the classic Japanese song, Zankoku na Tenshi no Tēze by Yoko Takahashi.<h3>The Voice of Our Team</h3>As the day came to a close, we collected feedback from our team. Many highlighted the retreat as a good moment for team cohesion. "It was a lovely event and I think especially for people that just joined like me it was a great team building event. I got the chance to talk in person to many collaborators that otherwise I would have only met virtually" shared one team member. Another remarked, "The location was a nice change of pace, the award ceremony was really wholesome, and the after-party + karaoke were a blast!"<img src=/content/images/image_c4a3980016757f3a557d76efdd18bf16.jpg>“Take my picture as if I am not aware.” <h3>What’s Next?</h3>Our one-day company retreat was more than just a series of activities; it was a kick off for the journey that awaits our team for this year. A journey that not only reminds us the excitement of our goals, but also the undeniable strength that lies in our diversity that will help us overcome the myriad of challenges that we will face along the way. Here's to Recursive and to many more success!<img src=/content/images/image_fdd9bfc5504a544737d5af0f152b0ca3.jpg>Recursive team, more diverse than ever.

Recursive had a company retreat in March 2024. This blog provides a report of the event

Event Report: Recursive Holds Company Retreat to Accelerate Value Creation

Recursive is a service provider that offers AI solutions for building a sustainable future. As a Japan-based startup, it develops its own AI products. In order to always be able to propose optimal solutions to our clients' increasingly complex business challenges, Recursive recognizes and promotes the importance of connecting with researchers and students within the industry so that we can continue to stay ahead of the latest technologies and trends in AI, particularly in the field of natural language processing.Against that background, we had the pleasure of welcoming a group of students from Chalmers University of Technology in Gothenburg, Sweden, to our office in Tokyo!The students, part of the WASP (Wallenberg AI, Autonomous Systems and Software Programming) research initiative, visited Recursive’s office on March 13, 2024. Their focus on Machine Learning (ML), Artificial Intelligence (AI), and autonomous systems aligned perfectly with Recursive's core areas of expertise.<h3>Exploring Shared Expertise and Knowledge</h3>The day was a moment to exchange about our expertise and knowledge as we both presented our projects and engaged in lively discussions. The presentations made by Chalmers students, Adam and Lena, tackled pressing issues in machine learning, using complex methods to boost model effectiveness in practical settings. Adam Breitholtz's talk on transferability was particularly pertinent, addressing how it can overcome data scarcity and high data acquisition costs. Lena Stempfle's presentation delved into the intricacies of healthcare data, showcasing how machine learning can significantly enhance decision-making in healthcare, reflecting a profound grasp of the field's opportunities and obstacles. <h3>Looking Forward: Building Connections</h3>The visit wasn't just about research presentations. We fostered a casual and interactive environment where students and our team could connect. This allowed for a deeper understanding of Recursive’s work culture and the exciting tech landscape in Japan.<img src=/content/images/image_8746ee32622d17a2abf3c799c4cd7146.jpg>After the presentations, discussing about our workOne of our guests provided a positive feedback on how Recursive provided a welcoming atmosphere for this exchange event:Excerpt comment from one of our participants:“The mingling session after presentations allowed us to get to know the Recursive and its employees better, learn about the startup scene in Japan and discuss about the sustainably perception and efforts from Japanese society, government and companies. This shows the shared interests between Recursive AI and WASP and could be the starting point for further exchanges.”It was also extremely encouraging to hear that the students thought Recursive has a clear vision and some of the key projects that were briefly presented seemed interesting and environmentally relevant. <img src=/content/images/image_925af63934a74580394e21f457ba4c59.jpg>Group picture with our team and Chalmers University students These interactions solidify our belief that collaboration between companies and academia is key to pushing the boundaries of what's possible in the field of artificial intelligence. This visit served as a springboard for future collaboration. We at Recursive look forward to working with researchers and academia to co-create new innovations.

Chalmers students visit Recursive to discuss AI and Machine Learning. Both sides presented projects, highlighting the importance of industry-academia partnerships for AI advancement.

Event Report: Recursive holds exchange session with Chalmers University of Technology, aiming for industry-academia co-creation

In this article, we provide a description of some of the key responsibilities of our MLEs, and a brief overview of some of the problems that our Machine Learning Engineers solve on a daily basis.<h1>The key responsibilities of a Machine Learning Engineer</h1>In most projects, some of the key responsibilities of our MLEs are:<ul><li>Conducting comprehensive data collection, often involving literature and dataset reviews to assess dataset availability and data collection methodologies.</li><li>Employing data visualization and analysis techniques to extract meaningful insights from the collected data.</li><li>Engaging in data engineering tasks, which often requires deepening your understanding of the problem, and analyzing the data in different lights.</li><li>Developing machine learning or deep learning models, a process that typically encompasses literature reviews, the formulation of modeling proposals, technical discussions with the team, model training, inference runs, and result evaluations.</li><li>Collaborating closely with our project managers to gain a comprehensive understanding of the client's needs and devising a strategy for a successful project delivery.</li><li>Thoroughly documenting your work and sharing your findings with the team.</li></ul>While the list provided is extensive, it does not encompass all the aspects of being a Machine Learning Engineer at Recursive. The role is dynamic and multifaceted, with each project having its distinct requirements and challenges.<h1>What Machine Learning Engineers do at Recursive</h1>Recursive collaborates directly with corporations who present us with challenges that can be addressed through data science, machine learning, deep learning, software engineering, or a combination of these fields. In addition to our client projects, we actively maintain and enhance our own products, such as FindFlow (<a href=https://www.findflow.ai/>https://www.findflow.ai/</a>) and Borealis (<a href=https://borealis.jp/>https://borealis.jp/</a>).At Recursive, Machine Learning Engineers are integral to every data-related aspect of solving our clients' problems, encompassing data collection, data engineering, analysis, AI modeling, and model deployment.Our MLEs have engaged in diverse projects, including developing product recommendation systems, forecasting from time series data, generating images and videos using generative AI, and tackling natural language processing challenges.<h2>Groundwater level prediction</h2><h3>Problem description</h3>We have joined forces with IHI and Sumitomo Forestry to leverage machine learning for the forecasting of groundwater levels in a forest in West Kalimantan. The prediction of groundwater level holds significant importance as it informs the formulation of fire prevention strategies. This is especially pertinent in the Indonesian region in question, where the forests contain abundant peat deposits deep underground, which are rich in carbon content and are highly flammable, making fire management in these peatlands an arduous task, and emphasizing the critical role of fire prevention and groundwater level management in that area.Our client currently manages groundwater level through the irrigation and drainage of specific zones, primarily relying on sensor-generated groundwater level measurements collected throughout the forest. However, there is a distinct advantage in transitioning to a predictive groundwater level management approach, and in expanding the groundwater level measurements and predictions to cover the whole terrain of the forest, instead of being limited to specific groundwater level sensor locations. The motivation to achieve that was the driving force behind our collaboration with IHI and Sumitomo Forestry. In particular, our MLEs had the goal of delivering a method to provide groundwater level predictions with high spatial and temporal resolution up to 7 days in the future for arbitrary locations within the region of interest.In the scope of this project, our clients have supplied a diverse dataset specific to the West Kalimantan region, spanning several years. This dataset encompasses various information, including groundwater level measurements, precipitation records at select locations within the region of interest, manually recorded elevation data, the geospatial distribution of canals used for irrigation and water table control, as well as the water levels within these canals.<h3>The traditional solution</h3>The traditional approach to solving this problem would be to define a physical model of the region, which involves a detailed description of physical properties of the region at a high spatial and temporal resolution (e.g. topography, peat depth, weather conditions, hydraulic coefficients), as well as a precise definition of the physical equations that govern the subsurface water flow in a simulator. Such approach, when successful, tends to have a high predictive accuracy. However, the main challenges with defining a physical model are:<ol type=1><li>Data collection for physical modeling is resource intensive, but it is crucial. If the physical properties are not precisely described at high resolutions, the model accuracy cannot be guaranteed.</li><li>A physical model is specific to a single region, and it cannot be generalized to other regions.</li></ol>Given these challenges, the cost of data collection, and our client’s expectation to quickly apply one solution to different regions in the future without a lot of additional effort, we have proposed to tackle this challenge in a more data-driven manner.<h3>Our solution and the work done by our MLEs</h3>Our MLEs have worked with leveraging the data collected by IHI and Sumitomo Forestry with machine learning methods.The first steps of the project involved careful investigation and analysis on the provided data. By collaborating with the client, our MLEs built a deep understanding of the data and the problem, which was crucial for them to proceed with:<ul><li>Developing methods to clean and process different types of data (e.g. determining the best method to deal with missing values and outliers).</li><li>Developing methods to combine data collected at different frequencies.</li></ul>By building on these steps, they then successfully applied different machine learning models to make predictions on the sensor locations.However, due to the sparse nature of the data, both temporally and spatially, and the presence of noisy measurements, developing data-only methods, by either adapting our machine learning methods or developing deep learning models, to extend the groundwater level predictions to the whole region of interest would not be the best approach.Instead of relying solely on data, our MLEs developed a unique fusion of machine learning and physics-based modeling. By including physics-based priors to a machine learning model, our MLEs managed to overcome the challenge of working with a small dataset, and they empowered our client to forecast groundwater levels for the entire specified area, up to seven days in advance, as depicted in the image below.<img src=/content/images/image_a859d98eceb60af597b4e5db9dba8fed.png><h3>Why our solution is better than existing solutions</h3>The advantages of combining physical priors and machine learning compared to data-only approaches are:<ul><li>Physical priors enhance data-based predictions, specially when the data is noisy.</li><li>Physical priors are specially helpful to guide the predictions where there are no sensors. They can help our model make reasonable guesses.</li><li>The addition of physical priors encourage the model’s predictions to be in accordance to the laws of physics.</li></ul>The advantages of our fusion approach compared to the traditional physical modeling approach are:<ul><li>It is not required to have a detailed and high-resolution specification of the physical characteristics of the region of interest to apply our model.</li><li>The fusion approach developed by our MLEs can be applied to other regions without additional modeling.</li></ul><h2>Borealis</h2><h3>Problem description</h3>To make informed decisions, having access to historical weather data at a high spatial resolution is crucial. For instance, when deciding the optimal locations for solar panel installation, it's essential to consider factors like the amount of solar radiation received in a specific area, often measured within a few square kilometers for industrial purposes. Additionally, understanding weather-related variables is critical for identifying regions susceptible to certain disasters, such as wildfires, and taking preventive measures.Today, we are fortunate to have access to vast amounts of data. The Japan Meteorological Agency (JMA), for instance, has collected decades' worth of weather data, including information on solar radiation, precipitation, temperature, and more, from weather stations across Japan and through satellites and other sensors. This data allows us to trace the historical weather conditions at these sensor locations. However, the challenge we at Recursive aimed to address with Borealis was extending this historical data to cover the entire Japanese territory at a high spatial resolution.<h3>Our solution and the work done by our MLEs</h3>Borealis (<a href=https://borealis.jp/>https://borealis.jp/</a>) is our exclusive AI-powered product—an advanced dashboard empowering users to visualize various weather parameters (e.g., solar radiation, rainfall, temperature) across Japan in high-resolution maps. These maps are constructed using 20 years' worth of data gathered from weather stations distributed across the country. A concise overview of Borealis is provided in the diagram below.<img src=/content/images/image_b8d8148771b89514ffbcb993dade271a.png>Our AI model harnesses the combined capabilities of satellite, weather, and topographical data to extrapolate sparse weather measurements in Japan into high-detail weather maps with a remarkable 100-meter resolution.To accomplish that, our Machine Learning Engineers (MLEs) undertook the following tasks:<ul><li>Conducted comprehensive data collection from various sources, involving extensive research on potential datasets and, when necessary, the development of code for data retrieval.</li><li>Processed and analyzed the acquired data, which included cleaning and handling missing values.</li><li>Devised techniques to align data obtained at different times and locations effectively.</li><li>Researched and developed a deep learning approach to generate a high-resolution mesh from the sparse data points, covering the entire Japanese territory.</li><li>Together with our Software Engineers, enhanced the data and model pipeline, ensuring it could operate efficiently in production via API access.</li></ul>In the ongoing development of this product, our Machine Learning Engineers continue to collaborate closely with our Software Engineers. Together, they address various challenges, including data collection from diverse sources, data preprocessing, optimization of model inference speed to enhance user experience, and the integration of physical priors into the AI models. As the project evolves, new challenges may arise, and our team remains dedicated to overcoming them to further enhance Borealis.<h3>Why our solution is better than existing solutions</h3>Borealis offers several key benefits:<ul><li>Exceptional Map Resolution: Our generated maps boast significantly higher resolutions than those of the majority of our competitors.</li><li>Utilizing Decades of Reliable Data: Borealis leverages extensive datasets accumulated over decades from reputable and trusted institutions.</li><li>Empowering Sustainability Decision Makers: Borealis is dedicated to equipping decision makers with invaluable insights for their sustainability-driven initiatives.</li></ul>

In this article, we provide a description of some of the key responsibilities of our MLEs, and a brief overview of some of the problems that our Machine Learning Engineers solve on a daily basis.

The Machine Learning Engineer (MLE) role at Recursive

At the internal seminar of Scope Co., Ltd., our COO, Katsutoshi Yamada, delivered a seminar, which we will summarize in this digest. The lecture was planned to be divided into two parts, and this time we will cover the first part, which is the basics.<h3>Current situation: Japan is the world's second-largest plastic waste producer</h3>Japan has a high consumption of plastic products, and when comparing the per capita disposal of plastic containers and packaging waste by country, Japan is second only to the United States. When plastic waste is not properly disposed of and flows into the ocean, it not only affects the marine ecosystem but also causes pollution in coastal and marine environments. There is data that suggests it takes 400 years for a PET bottle to decompose in the ocean, and if the amount of ocean plastic waste continues to increase, it is said that by 2050, there will be more marine debris than fish. In Japan, disposable plastic containers and packaging materials provided in supermarkets and restaurants are considered a major cause of plastic waste. Efforts are needed to reduce these plastic wastes.To solve these plastic waste problems, Japan has implemented initiatives such as charging for plastic bags and promoting the use of reusable plastic containers. However, in order to further enhance environmental protection and sustainability, more and more companies around the world are working on this issue, while still maintaining it as a business.<h3>World business examples that combine sustainability and profitability</h3>First of all, the footwear and apparel brand Allbirds, founded in the United States in 2016, uses environmentally friendly materials as much as possible to produce its products. It has already gone public and is valued at over 180 billion yen, and is a typical example of building a revenue pillar while considering environmental impact. There are also fashion brands such as Ecoalf, which originated in Spain, and KAPOK KNOT, a Japanese apparel brand that uses materials derived from nuts, and more and more companies and brands are focusing on sustainability.Recently, there has been a growing interest in the technique called upcycling, which involves reusing useless byproducts and waste to upgrade them into more valuable new materials or products. There are specific measures such as upcycling beer lees into granola."Ethical Spirits" is a Japanese startup that uses upcycling to manufacture spirits from sake lees, for example. It has been talked about as the world's first recycling distillery that specializes in ethical production and consumption, and its packaging is also distributed as a gift because of its high design quality. In addition, the use of vegan leather products is increasing worldwide, in addition to apple and pineapple leather, cactus leather sneakers have also become popular.Furthermore, there are more and more companies with technology based on scientific discoveries obtained through research in specific natural science fields, which is called "deeptech". This concept, which is prevalent in Europe, is attracting attention in Japan as well. Hello Tomorrow, which has a global network of deeptech companies, entrepreneurs, researchers, and others, is promoting deeptech with companies and entrepreneurs around the world.<h3>World consumer trends</h3>In Europe, more than 80% of people are aware of environmental and social negative impact products, and they do not buy them. Beef is particularly recognized as a food that emits a large amount of greenhouse gases and some universities in Europe are not serving beef as a result. Worldwide, there is a growing awareness of avoiding products from companies that harm the environment, and more than 80% of people tend not to choose such products. In particular, there is an increasing interest in sustainable products and services.ESG (Environmental, Social, and Governance) investment is becoming more prevalent in universities and companies. The amount of investment in ESG is increasing every year and is expected to increase further by 2025. To catch up with the trend in Europe, companies in Japan need to focus on ESG. Although it is currently difficult to manage a business based solely on ESG, it is necessary to respond to it because there is also a possibility that money will no longer be collected without ESG.<h3>Government trends in Japan and the world</h3>The Japanese government has announced a policy to achieve carbon neutrality by 2050. Countries and regions aiming for carbon neutrality by 2050 are increasing worldwide, with the UK and the EU setting different numbers at the midpoint. It seems that the world is moving towards sustainability, and companies and consulting firms are moving in unison based on the government's announcement. There are many acquisitions related to sustainability, and major companies are increasingly creating sustainability departments, with more Chief Sustainability Officers being appointed. Experts in these fields are being recognized due to the high demand for sustainability, and companies are aware that it is important to communicate information about sustainability.<h3>The beginning of ESG management</h3>Since around 2020, there has been a surge in ESG (Environmental, Social, and Governance) investment. 30 leading asset management companies jointly established the Net Zero Asset Manager Initiative in December 2020, and in 2021, assets equivalent to 87 trillion dollars have joined the initiative, increasing the amount of assets under management. BlackRock, a leading institutional investor, announced a series of action plans in January 2020 to enable its clients to invest sustainably in the future. ESG investment as a whole has grown at an astonishing rate worldwide, with ESG investment increasing by 15.1% from 2018 to 2020, reaching 35 trillion dollars. It is expected to reach 53 trillion dollars, or about one-third of total assets under management by 2025.The movement in the advertising industry is further igniting this trend. CO2 measurement and reduction are now being demanded in the production and deployment of advertising. The organization Ad Net Zero has been established to lead the advertising industry towards net-zero, and companies such as Assembly have announced plans to launch tools to audit the CO2 emissions of the advertising industry. The Cannes Lions International Festival of Creativity has changed submission requirements this year and recommends that applicants submit information about CO2 emissions and sustainability impacts associated with their works.Although this is the first attempt, it is not included in the judging criteria, but it is important to understand these trends and develop business strategies that incorporate sustainability in the advertising industry.<h3>Voices of seminar participants</h3>We also received the following comments after the seminar.
”I felt the difference in the way of thinking and approach between Japan and the rest of the world. I was surprised to see how much of a business opportunity sustainability is.”
”I was surprised at how many business opportunities there are in sustainability. I was surprised at how many business opportunities sustainability offers. I learned a lot by switching my awareness from global environmental issues to a business mindset.”
”I was not clear on how sustainability activities are connected to business before the lecture, but now I understand better where the money comes from and how it becomes business.”Recursive will continue to promote the compatibility of sustainability and business in society.

COO Katsutoshi Yamada spoke at an in-house seminar for Scope Inc. This blog provides a digest of the seminar.

“Future-oriented Sustainability x Business Perspectives" Global Trends You Need to Know to Balance Sustainability and Profitability

From May 28th to June 2nd, 2023, I went on a business trip to three cities in the Middle East for the first time in my life: Saudi Arabia, Abu Dhabi, and Dubai.In conclusion, I returned home in a very excited state. I strongly believe that we should expand our business in the Middle East. I would like to summarize my experiences for other Japanese startups to consider. Below are the three reasons that I found particularly attractive.<h3>1. An overwhelmingly growing market</h3>The Middle East has set an ambitious goal to double its current economy size within 10 years. The engine for this growth is non-oil business areas, which account for 73% of the entire economy. There is an acceleration of investments in various fields, and new business opportunities are expanding.<h3>2. High interest in Japanese companies</h3>I felt that both the government and private sectors in the Middle East have a high interest in Japanese companies. There is a need to quickly progress in business. Moreover, as the presence of Japanese companies is not yet very high, there are opportunities for new entrants to stand out in the market. Especially in the field of AI, there is still relatively low competition, which can be described as a blue ocean state.<h3>3. Well-established support system</h3>The support from Japanese government agencies and JETRO is also well-established. They strongly support entering companies by connecting them with companies and government agencies, introducing them to events, and providing various information.Based on these elements, we can say that the Middle East market has great appeal for Japanese startups. For companies considering entering the Middle East in the future, I recommend using the support measures provided by the Ministry of Economy, Trade and Industry and JETRO, as well as researching the startup programs being deployed by the UAE government. This will help gain a deeper understanding of the business potential in the Middle East market.<h3>Additional inspection details</h3>I will also briefly introduce my experiences in each city. First, I visited Saudi Arabia. The first thing I noticed when I landed at the airport was that I had come in shorts. I immediately felt anxious about the cultural difference that one should not show their skin. There was a big gap between westerners wearing suits and my attire. I was also surprised by the abnormal heat and the feeling of sand in Saudi Arabia.I stayed at a Holiday Inn, which was more comfortable than a famous hotel in San Francisco. The quality of the hotel bathroom was also higher than expected, and the breakfast was impressive. I realized that there is no culture of exchanging business cards in Saudi Arabia, and I thought that QR codes should be used instead of business cards, not only for us but also for Japan.On the second day, we presented Borealis, which we developed, to the Saudi Arabian Energy Agency (photo) and received a lot of interest. I will write more about this later as it progresses.<img src=/content/images/image_3b90f505689ccceaed9445db56b93a42.jpg>On the same day, we also visited an exhibition of "THE LINE," one of the NEOM projects promoted by the Saudi Arabian Crown Prince. The scale of the idea was so big that it was not easy to express my thoughts about it.<img src=/content/images/image_98c3aff4180771450c33279489d72ff1.jpg>Next, I visited Abu Dhabi. The atmosphere, beautiful buildings, and townscape reminded me of Singapore. Compared to Saudi Arabia, I felt that the heat was somewhat calmer. I visited Hub71, which is jointly operated by WeWork and the Abu Dhabi government, and heard various stories from a Japanese startup based there. I directly felt the fun of working with diverse sectors and people.<img src=/content/images/image_05955786bab564e33a8d2dc07d058390.jpg>On the second day in Abu Dhabi, I participated in an event called "MAKE IT IN THE EMIRATES" held at the Energy Center, where I listened to various talks and presentations by startups from around the world. Furthermore, on the second day, I had a valuable opportunity to exchange opinions with government officials, the Energy Agency, and local large corporate executives in a roundtable format, which led to the next step.<img src=/content/images/image_1f61990158ffcd226871d31b054b0872.jpg><img src=/content/images/image_7a794b78bce8ec2c66c2619372949ec2.jpg>Lastly, I visited Dubai. I felt that the atmosphere was somewhat similar to Las Vegas. I met with the Vice President of Sobha Limited, a real estate development company, and was surprised by the scale and quality of the town they are developing. The completion of those projects is progressing as planned, and they are selling like hotcakes.<img src=/content/images/image_5ad5011259d800b7bf4b119da8c9d3bd.jpg><h3>Conclusion</h3>I am very excited to have experienced the attractive market of the Middle East. As I have chosen to be a startup, I have the desire to do something that can only be challenged here, and I think it should be done. I am aware that it will not be easy, but I hope to move forward with various help. I would be happy if this could be useful as a reference for Japanese startups.

This is a business trip report for the end of May 2023 to Saudi Arabia, Abu Dhabi, and Dubai. Publishing this report in the hope that it will provide some hints for your business in the Middle East.

Middle East Visit Report / The Potential of AI Business

Hello, this is WoojungThank you very much for the overwhelming feedback I received from many of you after my first post! Here goes a second one on Beyond EXPO 2023 in Macau. Hope you enjoy!Beyond EXPO 2023 in Macau was held from May 10th to 12th, and numerous East and Southeast Asian companies developing cutting-edge technologies in the areas of healthcare and sustainability exhibited. At the event, Tiago Ramalho, CEO of Recursive, moderated the discussion of “How is AI Shaping Southeast Asia's Next Gen?"While the innovative growth of Large Language Models (LLMs) is making headlines in the media on a daily basis, there are many challenges to their implementation to the level that it addresses the social issues. In order to make the vision of "AI for society" happen in the near future, each country and region needs to promote implementation suited to their own circumstances. Southeast Asia lags behind in DX compared to East Asia. However, that is why there is a great potential to innovatively accelerate DX in the region by introducing AI from the beginning, which is why Recursive is focusing on the Southeastern market. The following is a summary of the characteristics of Southeast Asia and the potential of AI in Southeast Asia based on the content of the speakers.<h2>Linguistic characteristics of Southeast Asia and their impact on AI development</h2>The most important characteristic of Southeast Asia in AI development, especially in LLM development, is that all the countries are multilingual. While the majority of the Southeast Asian population has the ability to speak English on a daily basis, each country and ethnic group uses its own language in addition to English.Traditionally, existing tech companies such as Google develop their software and AI in English, which makes it difficult to localize their products for launch in other language countries. However, the bottom line is that this should not be a major problem.In addition to Natural Language Processing and speech recognition technologies, the people in the Southeast Asian countries have a high level of understanding the nuances, intonation, and other linguistic elements that affect communication beyond the meaning of individual words which is an important foundation for the development of LLM.Furthermore, from the perspective of corporate strategy, LLM can be used for customer engagement, etc., and the speed at which the technology can be deployed will be comparable to other regions.<h2>Social Challenges in Introducing AI in Southeast Asia</h2>While the development and implementation of AI in Southeast Asia has great potential for growth as described above, there are also several social challenges that could slow the spread of AI.<h3>1. Lack of Governmental Policies</h3>The market potential of Southeast Asia is maximized when the 10 Southeast Asian countries are combined. However, less than half of the ten Southeast Asian countries have not yet announced their national AI strategies, and the ten countries are hardly moving collectively. Only Singapore, Thailand, Malaysia, and Indonesia, the countries with the fastest growing economies, have announced theirs, and the rest of the region still have some work to catch up. In addition, with the exception of Singapore, some countries have low levels of trust among governments, businesses, and citizens, and many countries lack the infrastructure and data systems necessary to introduce AI, which could raise the hurdle for AI adoption.<h3>2. Distrust Regarding Data Provision</h3>Again, because of the lack of a legal system for data use, companies and the public tend to have a high level of distrust on providing their personal data. As a result, it is difficult to accumulate data necessary for AI development, which is a challenge for AI diffusion.<h3>3. Lack of education on AI</h3>In order for AI to spread, it is necessary for the public to understand what AI is, how to use it, and to some extent about its risks. Since awareness of AI is still lower in Southeast Asia than in other regions, finding engineers and product talent is a challenge. Startups from Southeast Asia often find talent from overseas (mainly China), and the low level of AI literacy in the country as a whole will be a major challenge for future growth.The ability to learn and take an interest in not only technical knowledge but also how AI is being used around you and where your data is being used from an early stage will have a significant impact on the development of AI human resources in Southeast Asia. Institutional support for the reskilling required of workers as technology develops will be necessary.<h2>What is inclusive and fair AI?</h2>Although we have picked up issues in Southeast Asia, there are some things that Japan can learn from Southeast Asia. In Japan, even though there is a lot of noise about AI in the media, many people feel that it is still a distant topic for them, and even when I work as an AI consultant, I hear people say, "I never thought our company could use AI.” Singapore is an example of social implementation of AI.From the example of social implementation of AI in Singapore, we can say that "good AI" has four major characteristics. (1) It is beneficial to society, (2) it is transparent, (3) it is provided equally to all, and (4) it is explainable.Of course, the difficulty of implementing AI in a relatively small population of 5.5 million people is different from that of implementing it in a country with 100 million people. Japan still needs to make many efforts to introduce AI, including social changes, infrastructure development, institutional guarantees and support, etc. However, if we learn from advanced cases such as Singapore and have the right direction, we may be able to achieve a society in which everyone can live happily and AI can be implemented. .P.S.As we have been accumulating AI use cases in Japan since last year, Recursive has been working toward overseas expansion this year, and we have been putting a lot of effort into events, such as speaking opportunities in and out of Japan, exhibiting booths, etc. If you have any good opportunities for us to join, feel free to reach out to us anytime.Note: This article is based on the content of the event.

Beyond EXPO 2023 in Macau was held from May 10th to 12th, and numerous East and Southeast Asian companies developing cutting-edge technologies in the areas of healthcare and sustainability were exhibited. In this blog, we report on a seminar held at this event.

Beyond EXPO 2023 in Macau event report

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.4, the final installment, is a continuation of “How I can use Large Language models in my business?”, with “6. Enabling custom AI systems” and “Conclusion”Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-01/>Part.1</a>Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-02/>Part.2</a>Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-03/>Part.3</a><h2>How I can use Large Language models in my business?</h2>Part.4 is a continuation of Part.2 and Part.3, we have put together a summary of the various applications of LLMs in the enterprise context. This overview will give you a sense of the types of problems that LLMs can help you solve, and the ways in which they can be integrated into your existing workflows.<h3>6. Enabling custom AI systems</h3>Large language models can help make the development of customized AI models more efficient and cost-effective by enabling the extraction of insights from unstructured data and feeding it to scientific-based systems. This can lead to optimized industrial processes, improved scientific simulations, better demand forecasting, and more. By automating some of the data extraction and processing tasks, using generative AI can also help reduce costs associated with manual labor and increase productivity.When it comes to scientific simulations, large language models can be used to extract and crawl data from public and private data repositories that can be fed into simulators to improve accuracy and reduce the computational cost of simulations. This can lead to more accurate predictions and better understanding of complex physical phenomena.By analyzing electronic health records and other patient data, a model could be developed to create personalized recommendations for treatment plans, medication dosages, and more. These recommendations can be tailored to specific healthcare concerns, such as chronic diseases or mental health disorders, and can help improve patient outcomes while reducing healthcare costs.In infrastructure, these models can be used to extract insights from various data sources such as traffic patterns, energy consumption, and environmental impact. This data can then be fed into a model that optimizes the cost and benefit of developing a certain piece of infrastructure, such as a new highway or a wind farm.<h2>Conclusion</h2>In conclusion, large language models have the potential to revolutionize the way businesses operate, from knowledge management and business intelligence to customer service and vertical-specific AI models. By leveraging the power of language, businesses can extract insights from unstructured data and make informed decisions that drive growth and efficiency. With their ability to understand complex patterns and dependencies in language, large language models are a powerful tool for unlocking the potential of big data and advancing AI capabilities. As these models continue to evolve and improve, we can expect to see their impact on businesses grow even more significant in the coming years.

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.4, the final installment, is a continuation of “How I can use Large Language models in my business?”,  with “6. Enabling custom AI systems” and “Conclusion”

The value of Large Language Models for enterprise Part.4

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.3 is a continuation of “How I can use Large Language models in my business?”, with “3. Automation for everyone”, “4. Processing unstructured data” and “5. Customer Support”Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-01/>Part.1</a>Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-01/>Part.2</a><h2>How I can use Large Language models in my business?</h2>Part.3 is a continuation of Part.2, we have put together a summary of the various applications of LLMs in the enterprise context. This overview will give you a sense of the types of problems that LLMs can help you solve, and the ways in which they can be integrated into your existing workflows.<h3>3. Automation for everyone</h3>Large language models have the potential to make some APIs obsolete by allowing people with no knowledge of code to automate common actions and connect different software tools without writing code. APIs, or application programming interfaces, are a set of protocols and tools for building software applications. They enable different software tools to interact with each other, making it easier for developers to create new applications and services.However, APIs can be complex to work with, requiring developers to have a deep understanding of coding languages and syntax. This can create a barrier for people who want to automate common actions or connect different software tools, but don't have the technical skills to do so.Large language models can help to overcome this barrier by providing a more intuitive interface for automating actions and connecting software tools. By using natural language commands, users can instruct the model to perform specific actions, such as sending an email, scheduling a meeting, or updating a database. The model can then take care of the technical details, such as interacting with APIs, so that users don't have to worry about the underlying code.For example, imagine a marketing team wants to automate their social media postings. Instead of writing code to interact with different social media APIs, they could use a large language model to automate the process. They could provide the model with a natural language command, such as "Schedule a tweet for 2pm tomorrow with this image and caption." The model could then take care of the technical details, such as interacting with the Twitter API, to schedule the tweet.By making it easier for non-technical users to automate actions and connect software tools, large language models have the potential to democratize the automation process. This could lead to a more efficient and productive workforce, as more people can take advantage of automation tools without needing advanced technical skills.<h3>4. Knowledge management and business intelligence</h3>Knowledge management and business intelligence are two areas where large language models (LLMs) can have a significant impact on enterprise operations. By using LLMs to analyze and summarize large amounts of data and content, companies can improve decision-making, streamline workflows, and drive business growth.Automatically generated summaries of documents and other content make it easier for employees to quickly access the information they need. For example, a law firm can use a large language model to summarize legal briefs or case law, making it easier for attorneys to quickly find the information they need to support their cases.Search and synthesis systems powered by large language models can be a game-changer for companies when it comes to accelerating internal work. By providing easy and efficient access to a company's internal documents, these systems can help any worker quickly find the information they need to do their job effectively. For example, imagine a marketing professional who needs to create a new campaign for a product. Instead of spending hours manually searching through past campaigns and marketing materials, they could use a search and synthesis system powered by a large language model to quickly find relevant information and even generate new content.One industry where search and synthesis systems have proven especially useful is healthcare. Healthcare providers deal with an enormous amount of data and documentation, from patient charts and medical histories to research studies and clinical trials. By using a search and synthesis system powered by a large language model, healthcare providers can quickly access this information and use it to inform patient care and research efforts. For example, a physician who needs to diagnose a rare disease could use the system to search for similar cases and relevant medical literature, saving valuable time and potentially improving patient outcomes.Another industry where search and synthesis systems have proven useful is industrial manufacturing. Manufacturing companies deal with large volumes of technical documentation, such as schematics, assembly instructions, and safety guidelines. By using a search and synthesis system powered by a large language model, workers can quickly find the information they need to build and maintain products, troubleshoot issues, and ensure safety compliance. For example, a worker who needs to repair a machine could use the system to quickly find the relevant schematics and troubleshooting guides.Finally, the retail sector is another industry that can benefit from search and synthesis systems powered by large language models. Retail companies deal with large volumes of customer feedback, sales data, and product information. By using a search and synthesis system, workers can quickly find the information they need to make informed decisions about product development, sales strategies, and customer service. For example, a customer service representative could use the system to quickly access customer feedback and complaints, enabling them to provide a better customer experience.<h3>5. Customer Support</h3>Large language models can be a valuable tool for companies looking to improve their customer service operations. Chatbots and virtual assistants powered by large language models can provide immediate assistance to customers, reducing the need for human support agents to handle routine inquiries. These chatbots can be trained to recognize common queries and provide relevant answers quickly and efficiently, resulting in higher customer satisfaction rates and lower support costs for the company.For example, a retailer can use a chatbot to help customers track their orders, check product availability, and process returns or exchanges. By training the chatbot to recognize common queries and provide accurate information, the retailer can reduce the workload for human support agents, freeing them up to handle more complex issues.One advantage of large language models is their ability to connect with a company's internal documentation, allowing them to provide up-to-date and accurate information to customer queries. With the correct access scopes, chatbots and virtual assistants can quickly search through a company's internal knowledge base and provide the customer with the information they need. This can significantly reduce the amount of manual work required to respond to customer inquiries, freeing up human support agents to handle more complex issues.In addition to improving customer service, chatbots and virtual assistants can also be used for internal communication and collaboration. For example, a company can use a virtual assistant to schedule meetings, set reminders, and send notifications to team members. By automating these routine tasks, the virtual assistant can help increase productivity and free up employees to focus on more important work.Continue to Part.4

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.3 is a continuation of “How I can use Large Language models in my business?”,  with “3. Automation for everyone”, “4. Processing unstructured data” and “5. Customer Support”

The value of Large Language Models for enterprise Part.3

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.2 includes “How I can use Large Language models in my business?”, “1. Accelerating Knowledge work” and “2. Processing unstructured data” Read <a href=https://recursiveai.co.jp/en/blog/the-value-of-Large-Language-Models-for-enterprise-01/>Part.1</a><h2>How I can use Large Language models in my business?</h2>Now that we have discussed the potential of large language models (LLMs) and how they are likely to become ubiquitous in the business world, you may be wondering how you can actually use these technologies to improve your own operations.To help you get started, we have put together a summary of the various applications of LLMs in the enterprise context. This overview will give you a sense of the types of problems that LLMs can help you solve, and the ways in which they can be integrated into your existing workflows.<h3>1. Accelerating Knowledge work</h3>Large language models can be used to accelerate the typical office worker workflow by being used as a "calculator for writing." This refers to the ability of large language models to quickly generate text based on a prompt provided by the user.For example, imagine a typical office worker needs to write a report on a complex topic. Instead of spending hours researching and writing the report, the worker can use a large language model to generate a first draft. The worker can simply provide the model with a prompt, such as "Write a report on the benefits of using artificial intelligence in business," and the model will generate a complete report that the worker can then edit and refine.Using a large language model as a "calculator for writing" can save office workers a significant amount of time and improve their productivity. Instead of spending hours researching and writing, workers can focus on other tasks that require their attention. In addition, using a large language model can also improve the quality of the writing, as the model can generate text that is well-written and coherent.It's important to note that using a large language model as a "calculator for writing" does not replace the need for human input and editing. While the model can generate a first draft, it's up to the user to edit and refine the text to ensure that it meets their specific needs and requirements.In addition to providing a prompt for the large language model, users can also provide structured data from another program and let the model incorporate the language into the report. This can be especially useful for generating reports that require a combination of structured data and natural language.For example, imagine a sales team needs to generate a report on the performance of their products. They can use a spreadsheet program to generate the data and then provide that data to a large language model. The model can then use that data to generate a report that incorporates the data in a natural language format. The report might include insights on which products are performing well, which are underperforming, and what actions the team can take to improve performance.By using structured data in conjunction with a large language model, users can generate reports that are both accurate and easy to read. The structured data provides the necessary information, while the language model helps to generate the narrative around that data.<h3>2. Processing unstructured data</h3>Unstructured data is information that is not organized in a specific way. For example, a long paragraph of text is unstructured data because it doesn't have any specific format or organization.On the other hand, structured data is information that is organized in a specific way. For example, a table with columns and rows is structured data because it has a specific format and organization.Large language models can be used to "read" unstructured data and extract the relevant information in a structured format. For example, if you have a long document with information about sales, a large language model can be trained to identify the relevant sales data and extract it into a structured format such as a table. This can make it much easier to analyze and work with the data.This is particularly useful in supply chain operations, where there is often a large amount of unstructured data in the form of emails, invoices, and other documents. By analyzing this unstructured data, large language models can help to identify patterns and insights that might not be apparent from traditional data analysis techniques.For example, a large language model could be used to analyze emails and other communication between suppliers and buyers to identify potential bottlenecks or delays in the supply chain. It could also be used to analyze customer feedback and social media data to identify trends and patterns in consumer demand.In addition, large language models can be used to improve the accuracy of demand forecasting models by incorporating additional sources of data. For example, a large language model could be used to analyze news articles and social media data to identify trends and events that might impact consumer demand.Continue to Part.3

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.2 includes “How I can use Large Language models in my business?”, “1. Accelerating Knowledge work” and “2. Processing unstructured data” 

The value of Large Language Models for enterprise Part.2

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.1 includes “introduction”, “What is a Large Language Model?” and “Why looking at Large language models and generative AI now?” <h1>Introduction</h1>In recent years, large language models have become increasingly popular due to their ability to process natural language and generate text that is virtually indistinguishable from that written by humans. While these models were initially developed for use in natural language processing tasks such as language translation and sentiment analysis, their capabilities have since been extended to a wide range of applications.Large language models are a subfield of Generative AI, which is focused on the creation of artificial systems that can generate outputs that are similar to those produced by humans. This includes not just language, but also images, music, and other forms of creative output.Enterprises have only just begun to recognize the potential of large language models to improve their operations and enhance their offerings. These models can have an impact on knowledge work in virtually any industry, from finance and healthcare to marketing and customer service. From accelerating knowledge work to optimizing business processes, large language models have the potential to provide valuable insights and automation that can help businesses operate more efficiently and sustainably.In this blog post, we will explore some of the key applications of large language models for enterprises. We will discuss how these models can be used to accelerate workflows, automate common tasks, optimize supply chain operations, and improve sustainability. We will also provide specific examples of how large language models have been used in these applications to demonstrate their value and potential impact.<h2>What is a Large Language Model?</h2>Large language models are computer programs that have been trained on massive amounts of text data, typically crawled from the internet. They are trained using advanced machine learning techniques that allow them to analyze patterns and relationships in natural language data, and then use that understanding to generate new text or perform other language-related tasks.The process of training a large language model is resource-intensive, often requiring thousands of graphics processing units (GPUs) to train the model on vast amounts of text data for weeks or even months. During this process, the model is exposed to trillions of words, allowing it to build a deep understanding of language patterns and structures.The basic principle behind large language models is that they are trained to predict the next word in a sentence or sequence of text. This simple loss function is used to guide the training process, but it can lead to emergent properties of the model as it learns to model text dependencies over a long range. This means that the model can make accurate predictions about what words should come next, even when those words are several sentences away from the current position.The result is a powerful tool that can generate new text, answer questions, and perform other language-related tasks with a high degree of accuracy. Large language models have been used to improve machine translation, natural language understanding, and even generate creative writing such as poems and stories.<h2>Why looking at Large language models and generative AI now?</h2>The potential of large language models (LLMs) for the enterprise is enormous. With their ability to process and understand language in a way that was previously only possible for humans, LLMs can have an impact on knowledge work in virtually any industry. As the capabilities of generative AI continue to grow, we anticipate that in the coming years, every company will be using these technologies in a way similar to how email or spreadsheets are used today.One of the main reasons for this is that generative AI, including large language models (LLMs), has the potential to revolutionize many aspects of business operations. By automating repetitive tasks, providing new insights and recommendations, and streamlining workflows, LLMs can help companies improve their efficiency, reduce costs, and gain a competitive edge.In addition, the adoption of generative AI is being driven by advances in cloud computing and machine learning technology. As these technologies become more powerful and accessible, they are enabling even small businesses to leverage the power of LLMs to solve complex problems and gain new insights.Moreover, we are seeing a growing number of startups and technology companies developing solutions based on generative AI, which are making it easier than ever for businesses to incorporate these technologies into their operations. This is leading to a proliferation of tools and applications that are specifically designed to help businesses take advantage of the power of LLMs.

Featuring "The Value of Large-Scale Language Models in the Enterprise” which is a four-part series. Part.1 includes “introduction”, “What is a Large Language Model?” and “Why looking at Large language models and generative AI now?” 

The value of Large Language Models for enterprise Part.1

Innovation, put simply, is the act of solving a problem. This is what distinguishes innovation from research, which is the act of figuring out how or why something works. When you're innovating, you're solving a problem using a new tool or methodology that hasn't been used before, with the goal of solving it faster, cheaper, or more efficiently than before.There are many reasons for establishing innovation as an important goal on this increasingly fast path towards sustainability. Firstly, because sustainability is the right thing to do and innovation will allow us to become sustainable more quickly. But in and of itself, stating that something is "the right thing to do" is an intangible, nebulous concept - how do we convince people (and corporations) of this?Thanks to shareholder and customer pressure, organizations are transforming their business models and operations to save energy, minimize pollution, and combat gender, racial and ethnic bias. In the long-term, sustainable practices can reduce costs, increase efficiency, and create a better society for the world at large. As more organizations embrace sustainable innovations, it is becoming increasingly clear that sustainability is no longer just a trend, but rather an essential part of doing business. Key to this change process is innovation.Innovation is about solving concrete, real-word problems, and perhaps contrary to popular belief, innovation does not automatically require preceding research knowledge. You don't have to be an accomplished researcher or scientist to be a successful innovator. For example, with the advent of the steam engine, the innovation work preceded the understanding (or research) of the steam cycle; the cycle of thermodynamics that underlies the steam engine was only discovered much later. People, (who are in this case, innovators) simply realized that if you build a machine with certain parameters, it would work.It is vital to recognize, however, that research can and often does lead to further innovation; once we understood the mechanism of the steam cycle, we were able to create more efficient machines that used the steam cycle. Innovation and research often go hand-in-hand, but there is no requirement of one for the other.But how can innovation (whether it precedes or proceeds research) specifically aid our pursuit towards sustainability? Consider, for instance, lab grown meat. Startups such as <a href=https://upsidefoods.com/>Upside Foods</a> are successfully growing meat directly from cells and plant feed, reducing the amount of pollution and waste generated by meat farming, as well as reducing animal suffering resulting from industrial farming.Innovation is the act of taking an understanding of a topic or concept and applying it to a problem where it can be useful. So, even as technology changes, the definition of innovation doesn't change, though the pace and character of innovation definitely do. A hundred or two hundred years ago, innovation was mostly done by elite members of society because accessing knowledge was a privilege. You needed free time (and therefore, economic capital) to access books and libraries, and often you were limited to localized knowledge of your immediate neighborhood, your country, and what was available in your language. This meant that there were huge disparities in the ability to innovate between different countries and different social and economic classes.Today, with information technologies and the internet, the number of people with access to knowledge has increased exponentially. This has leveled the playing field somewhat and has made innovation more accessible. Although there are still limitations based on language, socioeconomics, and geographic location, these have been significantly lessened; this change in access has meant that smaller communities are able to innovate almost as quickly or sometimes even quicker than areas served by big corporations.Companies and corporations, however, are often guided by principles of short-term profit growth, which can conflict with the goal of sustainability. Corporate sustainability is a principle that tries to align these two goals. Businesses strive to be financially healthy and return a profit to ensure that their business remains solvent over the long-term. At the same time, it is important to ensure that the outcomes of business have a positive impact on society.Companies that lack sustainability tend to do poorly over the long run for many reasons; not merely in terms of optics and public perception but also in terms of legal and financial risks. Unsustainable companies tend to lack innovation and remain stuck in a particular market, making them vulnerable to disruption by innovators (even those companies who are initially operating in a monopoly). When income is guaranteed, at least to a certain degree, people tend not to realize the importance of innovation. Innovation requires a level of long-term thinking from company leadership - and often a certain degree of monetary input. The output of these "innovation investments" pay off both directly as revenue growth, and indirectly - there is a strong preference for employees to want to work for and contribute to an organization that is sustainable, leading to a more engaged workforce.Innovation is essential for improving sustainability and addressing the challenges facing our planet. By developing new technologies, practices, and solutions, we can create a more sustainable future and protect our planet for generations to come. Companies like Recursive are at the forefront of this effort. With projects such as predicting groundwater level in forests, generating high resolution weather maps, and improving healthcare outcomes, Recursive is making a real difference in the fight for sustainability. We can all play a role in supporting and promoting sustainable innovation.

Companies are looking for creative ways to meet their sustainable development goals. AI provides many solutions.

Is Innovation the solution to Business Sustainability?

AI enables the development of new technologies every day, and applying it to sustainability could be revolutionary. Many companies are seeking creative ways to improve their environmental, social, and governance (or ESG) goals, and leveraging AI provides many solutions.Increasing your organization's sustainability impact can also help boost your bottom line by saving money and increasing sales. Consumers are becoming increasingly cognizant of environmental and societal issues, which means that sustainability is vital to the success and longevity of your organization.<h1>How AI Can Help You Meet Your Sustainable Development Goals</h1>There are several ways <a href=https://recursiveai.co.jp/en/book>AI can help your organization</a> reach its ESG goals. AI can accelerate innovation by enabling faster research and development strategies. Researchers can easily access information using natural language processing (known as NLP) and text mining technology. This can drive progress for your company and achieve sustainable development goals.AI has increased productivity for organizations across the globe. They benefit from higher economic outputs that can lift people out of poverty and reduce pollution, raw material use, and even land use. Thus, improving productivity in these industries can have a major impact on improving their overall sustainability.Prevention and mitigation are other critical aspects. As we continue to see more disasters caused by climate change, deforestation, and even ecosystem pollution, it becomes increasingly important to adequately forecast potential problems and mitigate them. Predicting when misfortune will occur can help your organization prepare in advance.AI systems can help accelerate education and improve work conditions for workers at your organization. We know that AI systems cannot set goals and achieve creativity. Human input is still crucial when figuring out what problems must be solved. This allows organizations to direct algorithms to optimize specific areas.<h1>Why You Should Leverage AI in Your Organization</h1>AI can positively impact your ability to reach all your organization's sustainable development goals. A study published in the scientific journal <a href=https://www.nature.com/articles/s41467-019-14108-y>Nature Communications</a> found that AI technologies strongly impact nearly all of the 17 sustainable development goals. Although any new technology has the potential for negative impacts, the positive impacts significantly outweigh any harm they might cause.AI is necessary to deliver sustainable solutions and allow organizations to reach their sustainable development goals while reaping the economic benefits. A <a href=https://www.mckinsey.com/business-functions/quantumblack/our-insights/global-survey-the-state-of-ai-in-2021>McKinsey survey</a> showed that AI implementation offers significant benefits for organizations, including cost savings.<h1>How AI Has Helped Organizations Reach Sustainability Goals</h1>AI can unlock trillions of dollars for organizations across the globe. When AI is implemented and deployed with care, it can help achieve ESG goals, such as:1. Data center coolingOrganizations across the globe utilize AI to save time and money and to help achieve their sustainability goals. Take Google, for example; implementing AI into their data centers allows them to make a few <a href=https://www.deepmind.com/blog/deepmind-ai-reduces-google-data-centre-cooling-bill-by-40>percentage improvements in power efficiency</a>. This quickly translates into millions of dollars in savings each year. Additionally, they can operate their hyperscale data centers more efficiently, allowing for far larger outputs than local workstations.The company achieved this by training the DeepMind team on deep neural networks on data and enabling them to develop a more energy-efficient cooling system. This helped produce significant gains in energy efficiency and reductions in their carbon footprint and operating costs.2. LivestockIn the agricultural industry, livestock farms are largely responsible for emitting a lot of carbon dioxide. Even though there are many efforts to create meat alternatives, it's not yet a reality to mass produce these. Right now, the focus is on reducing carbon emissions by making a few efficiency changes, which could result in more affordable nutrition.The animals in livestock farms require constant personal care from humans, which is a lot of manual labor. Beijing Unitrace Tech developed <a href=https://www.washingtonpost.com/world/asia_pacific/facial-recognition-china-animals-farms-agriculture/2020/08/23/9808c710-d6fb-11ea-b9b2-1ea733b97910_story.html>a facial recognition software</a> that uses AI to monitor each animal's behavior and habits. The technology can also detect illnesses and unusual behaviors using a deep learning system. So far, this has only been developed for cows because pigs are more difficult to detect with facial recognition. Without the need for human intervention, this drastically reduces the time it takes workers to physically monitor their livestock.3. Harmful speech detectionAI technology can monitor language and detect hateful and harmful speech patterns on social media platforms that use NLP. For instance, <a href=https://ai.facebook.com/blog/ai-advances-to-better-detect-hate-speech/>Facebook uses NLP</a> to control harmful speech, including bullying, racism, and violence. Another major issue on Facebook is the spread of fake news, which leads to misinformation that destabilizes democracies.Facebook cannot use humans to moderate content that gets published, which is where NLP comes in. Facebook didn't stop there, though. Its research team also trained agents to sample novel training to prevent hate speech from spreading. Additionally, the system can self-supervise, meaning it can detect information outside the training data.It can feel impossible to meet your sustainable development goals manually. It is often not clear what to measure and how it relates to the issues society cares about. Leveraging AI is a key contributor to reaching your ESG goals. Not only will it save you time, but it will also help reduce your overall expenses.

How AI and Sustainability Go Hand in Hand

The agricultural industry has a sustainability problem. Not only is agriculture responsible for as much as <a href=https://online.maryville.edu/blog/importance-and-future-of-sustainable-agriculture/>30%</a> of current greenhouse gas emissions, but today's farming practices are also actively harming the environment farmers need to keep healthy if they want to continue growing crops long term. Massive corporate farms that take up huge expanses of land threaten biodiversity. At the same time, the regular use of chemical fertilizers, herbicides, and pesticides all contribute to the soil's deterioration. That's in terms of both its health and a more literal sense — modern farming practices significantly speed up the soil erosion rate.These practices also leave farmers even more vulnerable to the growing effects of climate change. Extreme weather events, such as floods, droughts, and unseasonable freezes, will only increase in frequency. With few protections in place, these events will continue to wreak havoc on the agricultural industry. Luckily, there is a sustainable path forward for farming, and it's in the form of localized mini-farms.The construction of mini-farms would be especially beneficial in countries such as Japan. With very little flat, arable land available to grow vegetables and a cooler climate that necessitates greenhouse gas use in the winter, farming is a costly business <a href=https://sfp.ucanr.edu/pubs/SFNews/95Nov-Dec/japanese_ag_obs/>financially and environmentally</a>. On top of this, Japan is highly vulnerable to extreme climate events, including typhoons, heavy rains, and extreme summer heat. So, a future of vertical, climate-controlled micro-farms should be a welcome one for the country.<h1>Why Smaller Is Better</h1><a href=https://medium.com/age-of-awareness/how-small-farms-can-sustainably-feed-the-future-45baf2ef6b4e>Small farms</a> are more sustainable and yield <a href=https://pdfs.semanticscholar.org/2b3c/36a7398c6bc0ed7ed351db6b67a28003d265.pdf>higher amounts of produce</a> relative to their areas than their large counterparts because of how small farms typically operate. Smaller farms are more efficient and less harmful to biodiversity as well. However, reducing the size of farms alone wouldn't be enough to solve the sustainability issues facing farming. Instead, the industry needs to think even smaller.Mini-farms are contained, vertically built facilities that yield small amounts of crops monthly or even weekly. The closed, controlled environment of these farms eliminates the need for chemical pesticides and herbicides and allows for the reuse of water — an important defense against droughts. The purpose of these farms is to serve only the local population. They also cut farming's contributions to greenhouse gases significantly.Of course, you might ask, if micro-farming is such a great solution, why hasn't it already been implemented? Although there isn't a single answer to this, one major element is simply the fact that the tech wasn't there to make it a viable solution for a long time. Now, however, it's finally time to tap into the full potential of the benefits of technology in farming through artificial intelligence.<h1>How AI Can Enable a Sustainable Future Through Mini-Farming</h1>There's a reason farms today rely on pesticides and chemical fertilizers instead of trying to grow crops through organic means — it's cheaper and easier to scale. To create a controlled environment that can grow quality produce at a highly efficient rate, you need systems in place that can analyze a wealth of data and act on its findings. In other words, you need AI and automation to make mini-farms work. For a long time, the future of AI in agriculture seemed like a pipe dream. That's no longer the case.You can already see <a href=https://www.japantimes.co.jp/news/2022/01/23/business/japan-farmers-sustainable-agriculture/>AI efforts at work in Japan</a> to make farming more sustainable. Through data analysis, farmers have been able to determine how many seedlings to plant and the best time to do so, reducing the need for special fertilizers and cutting the use of pesticides by as much as half. Takuro Sato, an organic farmer, is even developing a soil resistant to pests, using data analysis to determine the optimal compost ingredients for the land and an app to manage the temperature of the soil in each of his fields.When applied to mini-farming, AI can do even more. Algorithms can be employed to keep interior weather conditions at the most advantageous levels. Light, humidity, temperature — these can all be optimized by AI and managed through automation. AI will also continue to learn from each growth cycle and will be able to determine which breeds grow best in specific conditions. It will even be able to select the varieties that produce the tastiest and most nutritious fruits and vegetables.Food waste can also be a thing of the past, thanks to mini-farms and AI food waste management. Instead of growing a large number of crops per season and hoping most of it doesn't go to waste, mini-farms can take a more targeted approach. By applying demand forecasting to frequent, small-batch growth cycles, farmers can get much better ideas of which crops to produce during specific times of the year and how much is truly needed. Through this approach, farmers can have a food waste reduction roadmap that actually gets them where they need to be.In the past, micro-farming might have been an expensive, time-consuming process. Now, however, it has the power to be the more efficient and sustainable solution modern farming needs.

How can AI enable a sustainable future for the agriculture industry? By making it easier to implement mini-farms.

How AI and Mini-Farms Can Provide a Better Future for Farmers

Recently there's been a number of cases where a machine learning model was trained and deployed, and after deployment users realized that it was making predictions that were either biased against certain minority groups or just predictions that defy common sense. Such predictions can have obvious harmful consequences if not dealt with, and the damage done by such mistakes can derail an otherwise successful AI technological implementation program.If we hope to prevent these mistakes we need to take fairness seriously from the outset. This begins by identifying all stakeholders that could be affected by this technology's effects from the outset, at the planning and development stage, and making sure that they're involved in the process from the beginning through the development all the way through the quality assurance step.When it comes to data, we are all aware that data privacy is of utmost importance. Yet we've observed that many companies still don't have data management policies in place, and are not transparent about how they handle data. It's often difficult for users to understand what a certain company's data policy is, as it's buried under a long legal document that doesn't reflect the reality of their infrastructure. We believe that it's important to develop a simple, transparent data policy that helps customers make informed decisions about their data while at the same time increasing trust in the system.Budgeting enough time for quality control and inspection is also often overlooked. Not only do we need automated regression tests and to test on separate data that was unseen during model development, we also should have human inspection to test for edge cases. Being able to explain certain decisions the model makes also helps to allay certain fears. The field of model explainability is still very young and in some cases we don't have adequate answers for how to explain the models' output, but certain visualization techniques and mathematical guarantees, when coupled with extensive testing can provide reassurance that the model is unlikely to create a catastrophic mistake. Coupling machine learning models with logic-based systems is another way to provide safeguards for unpredictable scenarios. Crucially, we need to budget enough time for these tests, which often are overlooked in a rush to accelerate deployment.At recursive we believe that developing AI following such simple principles we can ensure that AI developments will have a positive effect on society. A recent publication "AI for social good" by leading AI researchers has suggested that AI development following certain recommendations will be a critical enabling technology for achieveing the SDGs.In "<a href=https://doi.org/10.1038/s41467-020-15871-z>AI for social good</a>" the authors outline the key principles for ethical AI development:<ol type=1><li>Expectations of what is possible with AI need to be well-grounded.</li><li>There is value in simple solutions.</li><li>Applications of AI need to be inclusive and accessible, and reviewed at every stage for ethics and human rights compliance.</li><li>Goals and use cases should be clear and well-defined.</li><li>Deep, long-term partnerships are required to solve large problems successfully.</li><li>Planning needs to align incentives, and factor in the limitations of both communities.</li><li>Establishing and maintaining trust is key to overcoming organisational barriers.</li><li>Options for reducing the development cost of AI solutions should be explored.</li><li>Improving data readiness is key.</li><li>Data must be processed securely, with utmost respect for human rights and privacy.</li></ol>We believe these guidelines form only the minimum set of requirements, and as we learn more surely we will only add to this list. Broadening the diversity of AI developers and including the broader community in AI development is critical to this effort.The deep long-term partnerships that are required to solve large problems successfully can only be achieved if all involved parties successfully build trust in each other.

An introduction to the principles that are important for AI development in terms of fairness and ethical considerations.

Fairness and ethics considerations

When training a model, a machine learning engineer will optimize the model's parameters in order to minimize the error on the collected training dataset. Success at this task however, does not mean the model will do well on unseen data. That's why it is common practice to test the model on a testing dataset containing data that the model has never seen before. Now, even if we see a good result on the test dataset, that doesn't necessarily mean that the model will do well in production.Data in production can differ from the data the model has been trained on in a number of ways. Different pre-processing techniques, temporal drift or different sources of data shifting the data subtly are just a few of the reasons why the data we've collected might not exactly match production data.For a successful deployment, there thus are a number of additional checks and tests we should perform to make sure we are deploying a robust system to the field. Just as a few examples, here are a few things to consider:<ul><li>A system to detect degradation in our model performance in real time and log data with high error or model uncertainty.</li><li>A system that can compare features in production data with expected data features to detect whether the environment has changed and we risk making incorrect predictions.</li><li>A system to capture novel data points that could be interesting to label and add to the training dataset.</li><li>Traditional rule-based systems to prevent outputs that could be significantly out-of-bounds for the problem at hand and prevent catastrophic failures.</li></ul>With these systems in place, we can implement a continuous monitoring and retraining system that can continuously update the model as we obtain more data. In this setting, it is crucial to have in place some way to keep a version control of the data so that we know on what data each model has been trained on. This helps us debug issues with model performance by checking whether they were trained with the correct data or not.If human labeling is necessary, we need a process by which we can efficiently label our new data. For example, a fast labelling platform and a way to track which data has been labeled and not; a set of guidelines for human labelers to define what to do in the case of ambiguous labels or to clarify vague definitions; and a verification procedure to make sure that the humans are not making mistakes themselves; as well as data analytics so that our researchers can check that that data set level statistics make sense.In realistic scenarios, certain labels are less frequent than others and we need to make sure that we balance our datasets. Then, we need to make sure that when we present our data to labelers, they can label the instances that we're more concerned with. For example, if we have a data set where we have 99% dogs and only one percent wolves we might not want to label 99 dog images for every wolf we label. It will be better to label only 10 dogs for every wolf to maximise the cost benefit of our labelling efforts.Finally, when deploying an AI system we need to consider the risk profile of mistakes. Mistakes are unavoidable, and even humans make them. In fact we've observed that in general a well trained AI model will make fewer mistakes on average than a human. However when an AI model does make a mistake, it can end up missing the mark much more than a human would, and that is a problem in terms of risk mitigation.So it is important to determine the level of autonomy that this system will have and how many safeguards as well as to what degree it needs to be tested before deployment. For example a recommendation for a video website probably can be deployed more quickly and iterated on much faster than an industrial control system, where a catastrophic error could cause a physical malfunction.Recently there has been a great push towards "explainable AI" systems. However, in our experience these systems don't really work as advertised, since state of the art AI models work precisely by performing calculations in a high dimensional space, and projecting them back into a low dimensional space for a human to visualize ends up over simplifying the model workings and are not enough to understand whether the model will correctly generalize in situations where it encounters very unfamiliar data. While there is value in visualization techniques for debugging and to have an idea of what the model is doing, we must do more if we want to prevent catastrophic failures.Instead, it is necessary to extensively test the model in unfamiliar situations, as well as present it with "edge cases", artificially constructed data points meant to cause the model to malfunction. For example, in the case of image classification, researchers have created a dataset of visually ambiguous images to determine whether the classification models were able to fail gracefully or would just make random predictions. This type of testing helped shape a new generation of more robust algorithms.

The third step to a successful AI implementation is deployment. Even if you get good results on a test dataset, it doesn't mean that the model will work well in production. Therefore, we will show you some examples of checks and tests.

How to lead a successful AI initiative: Step 3 Deployment

Often overlooked, the very first step in AI model development is to determine a combination of model and data that fulfills the objectives that we're looking for. This also involves determining exactly what will be output by the AI model and what objective is going to be optimized for.With the inputs and outputs of the model in mind, we have an idea of the dataset required to successfully train the model. The first step is to take stock of how much data we have right now and how much more data will be necessary to collect. If we do need to collect additional data there are various trade-offs to consider, namely how much does it cost to acquire additional data points, and how much it costs to label them (which could vary depending on the level of expertise required of the human labelers).Depending on these variables we can estimate the size of the initial training dataset, and this can be compared to the cost of developing different training techniques which may require more or less data. For example, data augmentation, regularization and transfer learning are techniques which make it feasible to train on smaller datasets, while techniques such as self-supervised learning may make it possible to train on unlabeled data. All these techniques however, will increase the time needed to develop the model.After considering the situation with the data and having decided on a modelling technique to handle it, engineers can start implementing the model architecture. That is, an algorithm that can take in data and an output, which could be a prediction or a decision. To train it, we perform an iterative learning process on the computer where the algorithm is fed the training data, its predictions are compared to the desired output, and an error signal is calculated. From this error signal, we can update the model's parameters until it minimizes the error.Once the model is trained it's very important to check that it hasn't memorized the data, and that it is generalizing in a way that is going to be helpful. So we need to have an analysis step where we calculate the model's results on a test set, with data it's never seen before. As well as calculate how robust it is to perturbations. And perhaps we need to optimize it by tuning the parameters or by training it again with a different architecture.During this iterative process it may happen that we realize that some of our prior assumptions were incorrect. At that point, we might decide that we need to change either the training data or the algorithm and train it again. That means we have another, higher-level learning loop at the human stage. This is very different from a traditional software development process where we can make a test, write the code and then we can immediately check whether this code we've written is correct.It is then crucial to understand whether we've budgeted enough time for this iterative development loop, which depends on whether this kind of model has been done before and we can take the learnings from another team, or we are developing something from scratch; as well as what are the tolerances for accuracy and robustness, which can make development time and costs much higher. In certain cases, if the way the model interfaces with the environment is controlled by a human or involves some kind of feedback loop, it might be acceptable to trade off some accuracy for a faster turnaround time in development.From the compute perspective, this iterative process can take a long time and training the model various times for different architectures, hyper parameters and so on will expend a very large amount of computational power. Additionally, the machine learning engineer will often need to wait for an experiment's results before proceeding with the learning loop. So when we're budgeting for our development process, we need to account for both the longer time for research and development as well as account for the compute cost to train the model multiple times.

The second step to a successful AI implementation is model development. Often overlooked, the first step in AI model development is to decide on the combination of model and data that will meet the objectives you are looking for.

How to lead a successful AI initiative: Step 2 Model Development

When it comes to scoping, it is critical to start from defining exactly what problem needs to be solved. Now, if you've been following the recommendations in this book, it should be quite clear what problem it is you are solving. However, in many organizations we've witnessed projects being developed and approved without anyone explicitly stating what problem is being solved and what are the concrete objectives to be achieved.Without concrete objectives, we cannot define a concrete metric to be optimized. In that case, it is often left to the developers to choose a metric to optimize for. And after a lot of time has been spent in implementation, the team discovers that this is not what the organization wanted.To avoid this, clearly state the problem. How will the final product look like? What will it do? How will it behave? What are its outputs. Exercise empathy. How will this product affect the members of our organization? Third parties? Create a mockup. This will allow you to anticipate possible side effects. Just as often as specifying what it should do, define what it should *not* do. What is a non-goal. What features can be left for further iterations. Think of the 80-20 rule.Avoid following the hype. While it is important to research the state-of-the-art technologies to choose the best possible solution for the problem at hand, do not be driven by what's trendy but by what is the most cost effective way to solve your problem. We've found that many problems can be solved with 20 year old statistical techniques, or may just require a very simple machine learning layer on top of a robust data collection and processing platform.Define milestones and evaluation points. You should have clearly defined 3 or 6 month milestones to evaluate whether the project is on the right track, and if not, kill it or pivot. This is only possible is the problem has been concretely defined and can be quantitatively evaluated. At the same time, if the project requires long-term research, make sure the milestones are abstract enough to provide leeway for experimentation (for example, instead of requiring a finished prototype, a valid research milestone could check for whether a specific hypothesis has been verified or falsified).Over-budget for compute. You will always need more computational resources than your engineers estimated. And even if not, you can always use those resources for further improvements of the initial prototype.Recognize that the devil is in the details. Often, more time is spent adapting a finished prototype for production than in the research and development of the original prototype. Data in production might be different than the dataset we've collected. Or production hardware has more limited computational resources and the models must be optimized to run correctly and with lower power consumption. It is crucial to budget enough resources for this deployment step, or risk external parties thinking that the technology does not have the promise that they thought.

In scoping, it is important to start by defining the exact problem that needs to be solved. This article describes scoping as the first step to a successful AI implementation.

How to lead a successful AI initiative: Step 1 Scoping

According to an awareness survey conducted in Japan 2020, nearly 80% of individual investors answered that they use corporate ESG initiatives as a factor in their investment decisions.Investment in ESG keeps growing, and its global scale was approximately $30 trillion in 2018, and <a href=https://www.bloomberg.com/professional/blog/esg-assets-may-hit-53-trillion-by-2025-a-third-of-global-aum/>Bloomberg</a> estimates it reaches $53 trillion by 2025.The Financial Times shows that the majority of ESG funds outperformed non-ESG funds in the last ten years.<h1>What is ESG</h1>ESG stands for Environment, Social, and Governance and defines the issues that corporates should address to achieve a sustainable society.<a href=https://www.unpri.org/sustainability-issues/environmental-social-and-governance-issues>The United Nations</a> defines that "Environmental" issues include methane and plastics. "Social" issues have human rights and labor standards. And "Governance" issues include director nominations and corruption.Private evaluation organizations score the level of ESG achievement, and investments based on that score are called "ESG investing."While the SDGs are goals for the nation and society, ESG focuses on issues that private companies can influence. Although the scope is smaller, their impact on society is enormous. Corporate ESG initiatives are crucial to the achievement of the SDGs.<h1>History</h1>The concept of ESG began to gain global recognition in 2006. The United Nations announced the Principles for Responsible Investment (PRI), an initiative aimed at institutional investors, arguing that ESG-based "responsible investment" is necessary to realize economic returns should accompany a sustainable society and that responsible investment.Coincidentally, the collapse of Lehman Brothers in 2007, a year later, led to an increased focus on corporate governance and a corresponding rise in interest in ESG investment.What triggered the spread of ESG investing in Japan was the signing the PRI by the GPIF (Government Pension Investment Fund) in 2015. When the GPIF, Japan's largest institutional investor, announced its participation in ESG investment, the momentum for ESG investment in Japan quickly grew.In 2021, Japanese Prime Minister Yoshihide Suga has pledged a 46% reduction in greenhouse gases by 2030.<h1>Value</h1>In 2020, under the circumstances caused by the coronavirus, <a href=https://www.spglobal.com/marketintelligence/en/news-insights/latest-news-headlines/esg-funds-beat-out-s-p-500-in-1st-year-of-covid-19-how-1-fund-shot-to-the-top-63224550>19 ESG funds outperformed S&P 500</a>. It shows how ESG works for risk management and performs well in volatile markets.<a href=https://www.morningstar.com/articles/991535/esg-investing-is-about-long-term-risk-management>Morningstar</a> shows some examples of long-term ESG risks:<ul><li>Governance: The creation of fraudulent accounts caused by prioritizing quarterly results over the interests of long-term investors.</li><li>Environmental: Greater regulatory risk, including higher taxes and fees imposed on emissions.</li><li>Social: The opportunity cost of failing to take care of employees.</li></ul>Some investors bet on ESG prioritizing social impact over financial outcomes.Many of them are younger investors who genuinely want to build a better working environment and a sustainable society for their retirement and their children's generation. In 2017, <a href=https://www.morganstanley.com/content/dam/msdotcom/ideas/sustainable-signals/pdf/Sustainable_Signals_Whitepaper.pdf>Morgan Stanley</a> found that 86% of Millennials are interested in sustainable investing.Recently, the concept of "impact investing," which combines social and environmental contributions with economic returns, has also gained attention.Impact investing supports projects that aim to create a better society and environment, but unlike donations, it does not compromise on economic returns. Social and environmental impact as well as financial return are set as KPIs and the investment performance is strictly measured.For example, in agriculture, investments in eco-friendly agriculture reduce environmental impact, create jobs, increase food self-sufficiency, and increase profits by improving productivity.As this trend grows, the value of ESG-based markets will continue to increase, creating a virtuous cycle for companies and investors working on ESG and society, and the environment.<h1>Challenges</h1>As for the challenges, first, there is no unified standard for evaluating ESG.Private evaluation organizations conduct ESG scores. According to the Global Sustainability Rating Initiative (GISR), as of 2018, there are 253 different ESG evaluation methods and 132 evaluation organizations.For example, MSCI has 10 themes and 37 factors it considers when developing a score for a company, while FTSE Russell has 14 themes, and S&P has 23 criteria.They have different evaluation methodologies, and as <a href=https://www.ft.com/content/df328c34-6d9b-4fe6-9074-74091ce23ac7>The Financial Times</a> points out, it's even possible that an ESG score from one evaluation organization is high, but one from another is low.This flood of evaluation organizations and methods has become an obstacle to corporate ESG initiatives. This situation can lead to "<a href=https://www.ft.com/content/9692adda-5d73-11ea-ac5e-df00963c20e6>ESG fatigue</a>," as companies have to prepare data for multiple evaluation agencies in conflicting frameworks and pay hefty consultant fees to improve their scores.<h1>Future</h1>In 2020, the U.S. Securities and Exchange Commission (SEC) formulated a standard framework for ESG information disclosure. In Japan, in May 2021, the Financial Services Agency (FSA) released a report requesting ESG evaluation organizations to explain their methods and rationale.AI is also effective in ESG evaluation. For example, NLP (natural language processing) collects information from the Internet that may be relevant to a company's ESG. Classification algorithms link the data to appropriate fields and calculate scores with machine learning.If AI systems were utilized further, companies would no longer need to scramble to collect and disclose data for evaluation organizations, and evaluation organizations, could reduce the vast personnel costs from analysts.More and more companies will be strengthening their ESG initiatives to keep up with the trend of ESG investment, which seems desirable from the perspective of increasing the value of ESG investment. However, if companies do not understand the essence of ESG and pursue only short-term profits, it will be difficult for them to gain long-term support from investors. In some cases, they may even fall into the trap of "<a href=https://www.cnbc.com/2021/04/23/what-to-know-about-greenwashing-in-sustainable-investments.html>greenwashing</a>" or something similar.Estimates from the United Nations Conference on Trade and Development (UNCTAD) and the current investment situation show a shortfall of $2.5 trillion in annual investment to achieve the SDGs. This shortage is a sufficiently realistic figure given the global scale of ESG investing ($30 trillion~), as mentioned at the beginning.To achieve an actual increase in corporate value, companies would need to sincerely address the social and environmental risks they face and proactively communicate their efforts to investors through ESG frameworks. Such actions will create a positive chain of events in the market and society, which will become a significant driving force toward realizing SDGs and a sustainable society.

In an awareness survey conducted in Japan in 2020, nearly 80% of individual investors said they would use a company's ESG initiatives as a basis for their investment decisions. We have therefore summarised the value, challenges and future of ESG investment, which continues to grow.

The value of ESG for investors and society

When we scope out an AI development project it will be impossible to predict exactly how progress will play out. Unexpected issues may arise, such as model performance not being good enough and that might be due to, for example, not having enough data in our data set or the model architecture isn't appropriate to doesn't generalize adequately. Or we might discover in our testing that while we have 99% accuracy, the predictions fail catastrophically in the remaining 1% of the cases.So with this lack of predictability, how can we most effectively manage an AI development project? While there are no industry standard best practices, we can share some observations we've made throughout years of experience as well as through discussions with many industry players.Define all stages in the project lifecycle from the outset. This includes scoping, data acquisition, modeling and deployment. This will allow you to budget enough time for the full project all the way through to final implementation. We've seen many projects start with a quick POC that delivers promising results, but get cancelled due to lack of resources once the scale of work that needs to be done to make it robust enough for production is clear.Budget much more than you think for data labelling. Quick prototypes can be developed with off the shelf datasets available from the internet or quickly generated by an engineer. For a robust deployment in the field that will deliver positive ROI, the model needs to be trained on orders of magnitude more data. You will probably need to collect additional data and label it with human effort.Set up a testing pipeline from the outset. This is more of an engineering tip, but should also be considered when planning. Machine learning models are brittle, and some bugs can be introduced that subtly break things (the model still trains, but its accuracy is now lower). A continuous testing pipeline, at the code as well at the training level, will help catch regressions more quickly.90% of the work happens after deployment. Remember that after deployment, your model is alive. It will probably need constant monitoring, data acquisition, relabelling and retraining.Build up from a prototype. Instead of tackling the full problem from the outset, break it down into minimum viable objectives. Develop a model for the lowest hanging fruit and build up from there. Not only will it be easier and faster to show results to management, it will help identify faulty assumptions early on and correct them.

So with this lack of predictability, how can we most effectively manage an AI development project? While there are no industry standard best practices, we can share some observations we’ve made throughout years of experience as well as through discussions with many industry players.

5 tips for successful AI project management

While sustainability has emerged as a key concern for businesses throughout the world due to an increased awareness of the toll that population growth coupled with mass consumption has taken on the environment, many businesses are discovering that sustainable practices also help the bottom line - by cutting costs and increasing sales with consumers increasingly aware of environmental and societal issues.Yet, many organizations struggle with driving technological innovation due to past initiatives which underperformed, or an unclear vision of what the ROI will look like.We have distilled our experience to a six point checklist that can be used by any decision maker in an organization to implement a technological change program that will result in a measurable positive impact in your organization's sustainability footprint.<h1>Step 1: Research key areas of impact</h1>The first point is "Research key areas of impact". As we mentioned before while discussing the SDGs one of the big problems is that while there are well defined quantitative targets, they are very difficult to map into specific contributions that a particular organization or department can perform. So the very first step is to start by researching areas of impact.Different businesses will have different areas of impact. For example, if you have a farming business then land-use and clean environment might be priorities for you. For a traditional professional services company, labor productivity and gender equality might be on top of the agenda. Going through the list of Sustainable Development Goals can provide an insight on what areas might be relevant for your company by highlighting areas in which your organization has an impact. While not directly related to the sustainable development goals, ESG task forces and standards boards are also relevant sources of information from which you can determine which metrics are most relevant to your organization.Having determined which goals are most relevant to your organization, it might be worthwhile to determine which metrics you can have the most impact on. Factors that might be considered include: the size of your organization; your organization's impact on this metric; and how easily you can leverage existing solutions to produce meaningful improvements in this metric.To be able to do this systematically, sketch out a proposal for the six points that we propose below and map out the metrics that you might be targeting. As well as what potential solutions could be used, scope out what is the estimated return on investment and what is the estimated impact on your organization of taking these measures.You can then use this list to gather feedback and buy-in from different stakeholders in your organizations around which area you'll focus on first. If your organization hasn't done any sustainability initiative before, you might consider limiting the scope of your program to one single area with the most consensus, to be implemented as a trial program. After implementation, the ROI of this initiative can be used to bring your organization's members onboard for further change programs.<h1>Step 2: Define key metrics</h1>Once we've defined which ones are relevant and not, we can look at each goal's targets and indicators. Those are the actual quantitative measures of our progress towards achieving this goal. Now step number two is to define a key metric. Often, the SDG indicator will not be directly applicable to your company so we need to define a metric that we believe will directly and positively affect that indicator. So as an example, let's say we decide to focus on goal number three, which is to ensure healthy lives and promote well-being for all at all ages. We can go through the different targets and indicators and pick those indicators that are relevant to our company, so for example if your company has a factory in a particular community we can identify that the indicator 3.9.1 "mortality rate attributed to a household in ambient air pollution" could be impacted by our company's actions.So a key metric we can define is what are the concrete PM2.5 or harmful gas emissions attributable to our factory? We can make a commitment to drive that number down ideally to zero and we can measure the impact of that which should directly translate to an improvement in the indicator we are interested in.Having defined a metric, you need to decide how to measure it. Depending on the type of metric, this data might be available in the form of existing corporate data, recorded in documents or spreadsheets. It might also be measured by sensors deployed in your physical facilities, or might need to be estimated following standard accounting procedures.It may seem that a metric for which you already have the data would be an easier target to measure. However, organizations often have trouble with information silos, and it might be inefficient to collect all the data from different scattered documents in your organization. That's where natural language processing tools could be of help.Recently, we've seen the development of several AI-based search tools which seek to unify the information silos in a company and aggregate all the information into a data lake, or a knowledge base. Such knowledge bases contain aggregate information on the company and might make it much easier to measure this data. If you have such a digital transformation initiative in your company underway already, these metrics might be much easier to measure than otherwise. Similarly, if you already have sensors in your physical locations, data aggregation might be easier, considering your organization already has protocols for data aggregation, statistics, and summarization.If not, the cost of deploying a network of sensors in physical locations, and the software development required to aggregate and process this data must be considered when accounting for the costs of such a sustainability project.Some of the targets cannot be practically directly measured, and must be estimated by scientifically based accounting methods. For example, to estimate a company's carbon footprint, the <a href=https://ghgprotocol.org/>GHG protocol</a> is a good source of a consistent set of standards. As they are being calculated by indirect means, improvement in these targets will require a more detailed plan as there might be many contributing factors that indirectly affect the outcome. As such, it might not be immediately clear where action is required. Thorough evaluation of the accounting protocol to determine all the inputs that factor into it will help you map those contributions back into tangible measurable business metrics.<h1>Step 3: Pick a solution</h1>While in this book, we are focusing on technological solutions to achieve sustainability goals. It bears mentioning that in many cases a non-technological solution might also yield significant improvements in sustainability. For example, changes in your company culture or certain business practices might be all that is necessary to achieve a significant improvement in goals such as gender equality, better pay, or even energy efficiency. Even in such cases, quantitative measurement and tracking will help drive progress.Assuming we've determined a technological solution is a key piece of our sustainability initiative, we need to pick the right technology to achieve our goal. To make an informed decision we need to gather more information about our problem.In our case we are primarily concerned with an AI-first technological solution. When developing such a solution, a few of the topics we need to have a deep understanding of before starting a project are:<ol type=1><li>Data. Do we collect enough data to measure the metric we have defined in step 2? If not, how can we measure it?</li><li>Technology status. After consulting with experts, what does the existing technology allow us to do right now? What will it allow us to do in 5 or 10 years? What are the productivity and cost tradeoffs of the different solutions available.</li><li>Milestones. Considering the technology available now and any technology we can develop in the future, what are concrete yearly or quarterly target milestones for our metric?</li><li>Side effects. Are there any potential unintended consequences of implementing the technology? Have we considered all the stakeholders impacted by this change? Is this project sustainable in the long run - economically as well as politically?</li></ol>A full technological solution could require a combination of physical tooling deployment, software and AI model development, or development of a completely new process to replace a legacy system. In the case of hardware development and deployment, we need to consider fabrication costs and quality, as well as in-field reliability and maintenance. In the case of software development, data intake and output, deployment requirements (for example cloud versus on-device) and integration with existing systems must be considered.To prevent unintended side effects it is critical to not only collect information from all stakeholders but also to explain in detail what is the goal of this plan, why it is important and what steps will be taken. This enables all potentially affected parties to raise any concerns that might be overlooked by the original planners. This step also makes the transition process later on much easier, as everyone will be aware of the incoming changes and will have had a say on it. This is a key point as in many organizations the main hurdle to technological adoption and innovation is political, in the form of marginalized individuals or departments blocking a change because they were not involved in the decision process.It is also critical to consider the post-deployment stage. Have we calculated the total cost of running the system? Is its maintenance and upkeep sustainable from the economical as well as the human resource standpoint? To keep a system running optimally it might require updating to keep up with the reality of changing environmental, political and business conditions.<h1>Step 4: Development</h1>Now, picking a technological solution is not the end of the story. While there is a lot of hype around AI solutions in the media, developing an AI system successfully is not an easy task. AI and machine learning model development is a very different paradigm from traditional software development, and is still an early field with not so many established practices in the industry.The first thing to keep in mind when we are developing an AI project and specifically thinking about the whole life cycle of the project is data. This starts at the point of defining what is the metric we are measuring, which if you're following the task list that we propose you will have. Once the metric is defined we need to understand how data is going to be collected, processed and stored, not just for model training but also for inference at deployment time and performance evaluation.The model development itself is an iterative and experimental process, which requires adequate time and computational resources to be budgeted. Consider the tradeoffs between different solutions. Is it better to adapt an established technique to our use-case? Or are we researching a novel technique? The project management that is required for those different strategies is completely different, and we will cover the techniques necessary for a successful project in a later chapter.<h1>Step 5: Deployment</h1>Next we need to deploy our new solution. While there are technical challenges with respect to integration with legacy systems or unforeseen bugs in production, we found that the toughest barrier to deployment is societal and political. Humans are adverse to change, and we need to make sure that the change is not only beneficial but also well understood. This is why we believe that fairness should be a priority when developing AI systems.When it comes to this, a critical early action to take is to get the buy-in from other people in your company. Lack of enthusiasm for a project tends to be the number one stumbling block for most innovation initiatives. Present your plan early and communicate it to as many stakeholders as possible. Collect their ideas and integrate them into the plan as far as possible to increase a sense of ownership.Listening to people's concerns and possible objections might help you formulate a plan that is less likely to step on other people's toes. Moreover, by listening to concerns in advance it will be easier to implement mitigating measures for issues that you might not have realized in advance.Particularly when it comes to driving sustainability, we find that proposals can be presented with a sense of righteousness and that may put some people off as they feel being attacked. We propose instead focusing on the positive change that this plan can bring about, as well as a detailed ROI analysis, highlighting the value that implementing this project can deliver to the organization.Another possible area of objection centers around the robustness of the system. Fairly so, some stakeholders might object to deploying a novel technological solution that is unproven in the market, or where there are substantial risks to a botched deployment. To mitigate this it is important to follow the practices we outline in this book for safe and robust model development, and communicate them to the team.To summarize, to alleviate concerns around model deployment, focus on early communication with stakeholders, listening to concerns and integrating mitigations into the plan, as well as performing extensive testing before a wide rollout. In the fairness and ethics chapter we will also cover how extensive visualization of the model properties as well as taking the time to build in explainable AI might help allay concerns proactively.<h1>Step 6: Maintenance</h1>Finally, after deployment we cannot forget about our solution. In the case of a machine learning system, we often see the problem of data set and environment drift, which means the data that we're training our model on or the assumptions that we've made might change. The model predictions themselves might create a feedback cycle where the data that we're getting becomes different to what we saw before, so the system performance should be continually monitored and we should make provisions for a fast update cycle.In general with increasing speed of adoption of new technologies we see that we cannot rely on a static plan for many years, as our information about the world changes as well as the available technology. Therefore a successful technological implementation requires constant monitoring and re-evaluation to make sure it remains performant and relevant for a long time after deployment.

We have distilled our experience to a six point checklist that can be used by any decision maker in an organization to implement a technological change program that will result in a measurable positive impact in your organization’s sustainability footprint.

The 6 step plan for sustainable technological innovation

Material discovery is a research area with the potential to increase the sustainability of our economy in several ways. For example, by improving the efficiency of existing products and industrial processes, as well as enabling new applications and technologies that were not possible before.This is a crucial research field that often involves a lot of manual and time-intensive labor and trial and error by humans. Artificial intelligence could greatly help with this, by automating a lot of the work. We don't expect artificial intelligence to completely replace the end-to-end discovery pipeline, as we still need humans to guide its development and set the goals and understand the real world implications of certain developments. But we can see how it could help a lot with some aspects of the process.AI is beginning to gain a foothold in accelerating certain tedious parts of this system and automating experiments that often require trial and error.In the case of material discovery, there are two main areas where AI could be useful for the innovation pipeline: material property prediction and material synthesis.<h1>Property prediction</h1><img src=/content/images/image_4dcca4c8474db24414b750773186c4d3.png>When material engineers have candidates for a novel material, they need to determine whether that candidate will be suitable for its intended purpose.And to do so, a number of physical properties need to be characterized. For example, what is its pH? What is the temperature at which it reacts to other materials? What is its strength? What is the crystal structure of the material?And of course, it is also important to predict what will be its actual effect when integrated in its final application. For example, in the case of a battery, what is its ultimate energy storage capacity? How fast can it charge and discharge? How many cycles can it withstand before degradation?Predicting these properties often involves trial-and-error experiments, but by using data sets of other existing materials, we can <a href=https://aip.scitation.org/doi/full/10.1063/5.0018384>train artificial neural network models</a> that can generalize to other novel materials and can predict their properties both at the macroscopic level as well as at the microscopic level.For example, at the macroscopic level, we could simulate the bulk structure of this material to determine its last transition temperature, tensile strength, melting points, density, viscosity, and energy capacity, and so on. At the molecular level, we can also predict its microscopic properties, such as lattice constant, band energy, or electron affinity, which are crucial for development of faster semiconductors or batteries and novel chemicals.<h1>Material Synthesis</h1><img src=/content/images/image_fedf96259a688cc4a8140322f11dce18.png>In addition, artificial intelligence can be used to discover and propose novel materials from scratch. For example, by filtering which chemical compositions are likely to form compounds, we can develop systems that recommend element combinations from a pool of elements or present ionic substitutions in existing materials for the discovery of new compounds.We could also use these models to predict the crystal structure of a new material, so that we have a more guided exploration of which molecules are likely to have the crystal structure that we desire. <a href=https://aip.scitation.org/doi/10.1063/5.0022870>An example</a> from novel OLED systems shows that machine learning has been crucial to develop more optimized structures.AI can be also be used for process optimization. For example, to optimize the processing of material manufacturing, such as metal smelting. There are several key steps in this pipeline that could be accelerated by the use of AI, such as automating the analysis of the ore concentration, the processing of the ore, and the separation of the metals.For example, by developing a model that can predict what mechanical properties an alloy will have given its processing parameters, we can optimize the processing parameters for the optimal mechanical properties in this alloy, and thereby improve the smelting method.Additionally, AI methods have also gained a lot of popularity for the quantum mechanical simulation of these systems. When we're designing materials at the molecular level, quantum mechanical properties are very important and we need to have an accurate quantum mechanical description of the system. This is given by the density functional of the system, and often, these are very tedious and large calculations that have to be approximated numerically. And they may fail for very complex systems. However, by <a href=https://journals.aps.org/prresearch/abstract/10.1103/PhysRevResearch.2.033429>applying machine learning</a> to the estimation of density functions, we can more quickly and more robustly predict the density functionals with higher accuracy.

Material discovery is a research area with the potential to increase the sustainability of our economy in several ways. For example, by improving the efficiency of existing products and industrial processes, as well as enabling new applications and technologies that were not possible before.

Accelerating Material Discovery using AI

The field of modern medicine is ripe for disruption. Biological systems are complex, high-dimensional and hard to describe using traditional scientific methods. Up to now, progress has been slow since each step forward requires deep empirical investigation powered by human trial and error. The emergence of powerful computers is starting to revolutionize this field by enabling large-scale simulations of biological systems, as well as accelerating drug targeting and discovery based on historical data using AI algorithms.Drug development is an interesting case study that illustrates how AI is not a tool that can replace an innovation pipeline end-to-end. Rather, we need to consider the pipeline’s purpose and all its different steps in order to identify certain parts that can be replaced by AI systems. While incredibly powerful at discovering hidden patterns in data, AI algorithms are best applied to specific problem domains where a quantitative optimization objective can be defined.<h1>Drug development pipeline</h1>The drug development pipeline is an excellent example where we see AI’s pattern matching and high dimensional optimization complement the human ability to think creatively and direct discovery based on real world goals. If we break this pipeline down into its core components we find that there are a number of different steps that come into play for drug discovery.<h1>Target identification</h1>To develop an effective treatment for a disease, we need to find molecules that are relevant to that disease’s mechanism, and understand their structure so that we can design a drug that can attach to them and modify their behavior. For example in the case of a bacterial or viral infection, we would need to characterise molecules specific to those pathogens that the drug can attach to and how that drug might prevent their replication. This can be done in silico (computer simulation) but it most often needs to be confirmed in vitro, that is, a test system for the drug in a real live biological system.So how can AI help speed up this stage in the development pipeline? The very first step is target validation, which means that we need to identify targets in a biological system that could be disrupted by certain molecules; for now this is a human task that relies on the mechanistic understanding of the biology of the system, but our knowledge of biology is still very rudimentary. Unlike in other physical systems, it is difficult to isolate specific components of a regulatory pathway involved in a single process, because natural selection tends to create systems that are highly entangled and with a high level of reuse, creating very complex systems that are hard to characterize.To improve our understanding of this process, AI systems can be used to <a href=https://onlinelibrary.wiley.com/doi/abs/10.1002/widm.1068>reverse engineer the regulatory networks</a> in cells from experimental data. Using high-throughput assaying systems to measure multiple perturbed cell cultures, we can collect enough data variation and quantity to create a model of the regulatory network that is most plausible to have generated that data. Using that model, we can understand which perturbations in the network would result in a change in the disease progression.Once we know where in the molecular pathways we want to perturb with a drug, it is useful to know what is the molecular structure of the proteins involved. In many cases, we know the genetic code for the proteins, but we don’t know what is its final molecular structure inside a cell. It can be quite difficult to obtain this structure as it requires a complicated experiment where the molecules must be crystallized and then photographed with x-rays. In some cases this process damages the proteins and it can take years until researchers develop the right protocol to capture this structure correctly for a single protein.AI techniques such as <a href=https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology>AlphaFold 2</a> can greatly accelerate this process by predicting the protein structure from the genetic code. The model was trained on known genetic sequences and their corresponding protein structures, and can generalize to novel, unseen sequences. Even if not 100% accurate, these predictions can greatly accelerate the discovery process as experimental teams no longer need to wait years for each new protein to be characterized via crystallography experiments.<img src=/content/images/image_a9183da2353f779b0dbe8fe86f4505c9.jpg><h1>Molecular design and optimization</h1>We then need to find compounds that could potentially interact with the molecular pathways we’ve identified in such a way as to prevent or modify the course of a disease. Their efficacy must be tested empirically by synthesizing different molecules and testing them in silico or in vitro. After identifying promising drug candidates, it will be necessary to optimize their pharmacological properties to minimize side-effects and maximise their half-life in the body.The identification of candidate compounds can be sped up by a number of mechanisms:<ul><li>Molecule-based generative models can propose novel molecule candidates without human guidance. These AI systems learn to predict the structure of existing drug molecules and then generalize to predicting novel molecule configurations that have similar properties to known compounds. Such systems have been used to generate <a href=https://www.sciencedirect.com/science/article/pii/S0092867420301021>novel candidates for antibiotics</a>.</li><li>More directly, we can design <a href=https://www.nature.com/articles/s41592-019-0494-8>models that predict perturbations</a> at the single cell level without explicitly modelling the regulatory pathways. That means the machine learning model learns to model the cellular response to a certain perturbation by learning from a dataset of cell responses to similar molecules. With an accurate enough system, we can perform a quick computational screen of which compounds are likely to have an effect and which ones are not.</li><li>Some groups have also made progress in modelling drug and target interactions with less mechanistic understanding, relying instead on statistics. By leveraging data on existing drugs and their targets, it is <a href=https://pubs.rsc.org/en/content/articlelanding/2020/ra/d0ra02297g#!divAbstract>possible to train models</a> that predict the affinity of a given drug-target pair. Such models can be used to identify previously unexplored targets for known molecules, or generate novel molecule designs given a specific target.</li></ul>Beyond proposing candidate molecules for novel drugs, AI systems can also help in optimizing other pieces of the molecular design pipeline. For example, classification models can be used to predict pharmacological properties such as the half-life in the body or side-effects and drug interactions directly from the molecular structure.<a href=https://www.nature.com/articles/s41467-020-18008-4>Reinforcement Learning</a> can be a useful tool to create agents to automate the experiment design for in vitro testing, by automatically integrating experimental data and proposing novel configurations to test based on the historical experiment results.<h1>Clinical trials</h1>If a drug candidate shows success in in vitro and animal models, it can move on to the clinical trial stage which is broken down into three phases: phase I which checks for safety; phase II which tests for preliminary efficacy; and phase III where effectiveness in a large population must be demonstrated for approval. After drug approval, ongoing surveillance tests for any rare effects and risk/benefit trade-offs.AI systems can help accelerate clinical trials in a variety of ways. Improved data collection and record keeping systems using NLP and computer vision systems can prevent human error and accelerate data input. Adverse events can be immediately flagged and reported to prevent additional harm in the case of unexpected side-effects.Participant recruitment can be improved by ensuring that the population is well sampled, accounting for ethnic and other types of diversity. Genetic targeting and disease phenotyping using high-throughput sequencing and other technologies can be used to narrow down the range of participants to ones where the drug is more likely to have an effect, helping to show effectiveness more quickly in a smaller sample size. This kind of targeting might also allow us to confirm effectiveness in drugs that might otherwise be deemed ineffective by narrowing their use down to a more specific cohort, which is the promise of personalized medicine.<h1>Drug repurposing</h1>Given the vast number of effective compounds already on the market, a relatively cheap and efficient way to find new treatments is via drug repurposing. By using drug-target affinity models as we described above, we can identify potential new targets for existing drugs that are already known to be safe and not have major side-effects. This can shorten the discovery cycle as well as vastly accelerate the time needed to perform clinical trials for a specific drug.Drug repurposing has recently shown to be effective in helping advance drug discovery for the <a href=https://www.nature.com/articles/s41392-020-00417-y>recent COVID-19 pandemic</a>. A traditional drug research pipeline could not deliver results in time to respond quickly and effectively to an emerging epidemic, and drug repurposing using machine learning techniques has provided a lifeline to identifying promising therapies in a span of months, compared to years. While current efforts have had mixed results, a <a href=https://www.nature.com/articles/s43588-020-00007-6>recent study</a> has shown that lessons learned from this epidemic will greatly accelerate future drug repurposing initiatives.Overall, we believe that AI accelerated drug development has the potential to transform the field of healthcare over the coming decades with a faster experimentation and design cycle for more drug candidates, better understanding of the mechanistic processes of diseases and treatments, and more targeted personalized medicine becoming mainstream.

The field of modern medicine is ripe for disruption. Biological systems are complex, high-dimensional and hard to describe using traditional scientific methods

Speeding up drug development using AI

The<a href=https://en.wikipedia.org/wiki/Sustainable_Development_Goals> </a><a href=https://en.wikipedia.org/wiki/Sustainable_Development_Goals>sustainable development goals</a> were set in 2015 by the United Nations to guide our society towards a fairer, healthier and cleaner future.There are 17 generic goals that cover a range of areas from healthcare to clean energy to industry productivity. Progress towards each goal is quantified by a set of measurable targets. However, the UN has provided no guidance as how to achieve those targets.Governments are setting national roadmaps right now for societal scale projects that can help drive some of the targets forward, but this will not be enough. Some of the burden will fall on companies to innovate and adopt advanced technology in order to make their practices more sustainable.AI has emerged as one of the most powerful tools of the early 21st century. From<a href=https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/Driving%20impact%20at%20scale%20from%20automation%20and%20AI/Driving-impact-at-scale-from-automation-and-AI.ashx> </a><a href=https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Digital/Our%20Insights/Driving%20impact%20at%20scale%20from%20automation%20and%20AI/Driving-impact-at-scale-from-automation-and-AI.ashx>increased productivity</a>, to<a href=https://deepmind.com/blog/article/alphago-zero-starting-scratch> </a><a href=https://deepmind.com/blog/article/alphago-zero-starting-scratch>super-human performance</a> in<a href=https://ai.facebook.com/blog/rebel-a-general-game-playing-ai-bot-that-excels-at-poker-and-more/> </a><a href=https://ai.facebook.com/blog/rebel-a-general-game-playing-ai-bot-that-excels-at-poker-and-more/>various tasks</a>, AI can deliver solutions to problems that were before considered unsolvable. It is then fair to ask, What about sustainability?A<a href=https://doi.org/10.1038/s41467-019-14108-y> </a><a href=https://doi.org/10.1038/s41467-019-14108-y>recent study</a> has shown that AI will be a critical ingredient to achieve the majority of targets in 16 out of 17 goals. Companies that wish to meaningfully contribute to achieving these goals will have to implement AI solutions in their workflow.In this post we will show you specifically how AI can be used to drive measurable progress towards meeting the targets set by the UN.<h1>Higher industrial and agricultural productivity</h1>Higher industrial productivity allows us to maintain modern standards of living with less impact on the environment. The more efficient agro-industrial production is, the less land area and natural resources we use and fewer pollution we generate.In the agriculture business, AI is being used to<a href=https://www.theguardian.com/environment/2020/oct/08/behind-chinas-pork-miracle-how-technology-is-transforming-rural-hog-farming> </a><a href=https://www.theguardian.com/environment/2020/oct/08/behind-chinas-pork-miracle-how-technology-is-transforming-rural-hog-farming>monitor lifestock</a> at a large scale, improving productivity per animal; as well as automatically<a href=https://phys.org/news/2018-02-ai-pest-efficiency-environmental-impact.html> </a><a href=https://phys.org/news/2018-02-ai-pest-efficiency-environmental-impact.html>identify and remove pests</a>, which reduces the need for pesticides.Automated food sorting devices can be used to order food items according to consumer preferences, optimizing product quality automatically. These technologies rely on<a href=https://ai.facebook.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-/> </a><a href=https://ai.facebook.com/blog/-detectron2-a-pytorch-based-modular-object-detection-library-/>automatic instance segmentation</a>, a technology which has seen huge progress the last few years.In industrial production, instance segmentation and classification is also an important tool used in automated quality control inspection as well as in robotics. For example a robotic tool may be able to select specific parts from a bin or measure how many items are in a specific area using vision alone.AI can also be used for assembly line integration and optimization. Legacy systems consist of individual machines optimized for a specific task, with gaps in the process filled by humans. Using techniques such as<a href=https://en.wikipedia.org/wiki/Global_optimization> </a><a href=https://en.wikipedia.org/wiki/Global_optimization>global optimization</a> and endowing each tool with sensors, the assembly line can be completely automated and integrated.Forecasting techniques can be applied to historical data for predictive maintenance techniques. For example, we can predict when a tool is likely to fail and preemptively deploy a maintenance team. Resources can also be intelligently redirected by an AI controller in the case of a fault to prevent a system shutdown.Automated control systems powered by AI can also be used to<a href=https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40> </a><a href=https://deepmind.com/blog/article/deepmind-ai-reduces-google-data-centre-cooling-bill-40>optimize energy efficiency</a>, as well as<a href=https://link.springer.com/chapter/10.1007/978-3-319-47952-1_60> </a><a href=https://link.springer.com/chapter/10.1007/978-3-319-47952-1_60>reduce pesticide usage</a> by identifying which areas need to be sprayed or not. Similarly, food manufacturers can<a href=https://emerj.com/ai-sector-overviews/artificial-intelligence-industrial-automation-current-applications/> </a><a href=https://emerj.com/ai-sector-overviews/artificial-intelligence-industrial-automation-current-applications/>optimize their production processes</a> to use less raw material and reduce calorie counts in their final products.Logistics is another area ripe for disruption. A machine learning based<a href=https://arxiv.org/abs/2007.00882> </a><a href=https://arxiv.org/abs/2007.00882>algorithm</a> can predict public transit delays with up to 30% higher accuracy than existing algorithms, while a<a href=https://deepmind.com/blog/article/traffic-prediction-with-advanced-graph-neural-networks> </a><a href=https://deepmind.com/blog/article/traffic-prediction-with-advanced-graph-neural-networks>graph neural network</a> based system achieved up to 50% more accurate travel time predictions on Google Maps.These algorithms can be readily applied to global supply chains and logistics networks, potentially reducing their carbon footprint as well as speeding up transit times and reducing waste of perishable goods due to delays.Reinforcement learning is a promising technology to automate certain types of tasks that are hard to define specifically using rules or quantitive functions. This has the potential to<a href=https://ieeexplore.ieee.org/document/8675643/> </a><a href=https://ieeexplore.ieee.org/document/8675643/>revolutionize robotics</a> in the manufacturing industry by freeing up humans from repetitive, but infrequent, tasks.<img src=/content/images/image_eeeb013881ace88419618bbc10ac5ae1.jpg><h1>Accelerated innovation</h1>Many of the challenges outlined in the Sustainable Development Goals will require technological innovation to achieve. Climate goals, advance healthcare, and affordable food while conserving our planet's land and oceans can be achieved through scientific and engineering breakthroughs enabled by AI.In the field of healthcare, big data and AI can unlock the next generation of drugs via data-mining based<a href=https://hai.stanford.edu/blog/how-machine-learning-transforming-drug-discovery?sf132574259=1> </a><a href=https://hai.stanford.edu/blog/how-machine-learning-transforming-drug-discovery?sf132574259=1>drug discovery</a>. Techniques such as graph neural networks and attention have enabled breakthroughs in<a href=https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology> </a><a href=https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology>protein folding</a>, which will speed up drug research labs all over the world.In industrial applications too, physical simulations using AI can be used to deepen understanding of the<a href=https://deepmind.com/blog/article/Towards-understanding-glasses-with-graph-neural-networks> </a><a href=https://deepmind.com/blog/article/Towards-understanding-glasses-with-graph-neural-networks>physical properties of materials</a>, as well as<a href=https://phys.org/news/2020-12-machine-solution-materials-desired-optical.html> </a><a href=https://phys.org/news/2020-12-machine-solution-materials-desired-optical.html>accelerate the discovery</a> of completely new ones.Scientific simulation, crucial to the development of the next-generation of transportation vehicles as well as optimization of multiple industrial processes, can be accelerated by<a href=https://arxiv.org/abs/2001.08055> </a><a href=https://arxiv.org/abs/2001.08055>multiple orders</a> of<a href=https://arxiv.org/abs/2010.08895> </a><a href=https://arxiv.org/abs/2010.08895>magnitude</a>.<h1>Prevention and mitigation</h1>As natural disasters stemming from climate change occur more frequently and with more force, disaster prevention will become an increasingly critical area for policy makers.Based on historical data,<a href=https://ai.facebook.com/research/publications/riemannian-continuous-normalizing-flows> </a><a href=https://ai.facebook.com/research/publications/riemannian-continuous-normalizing-flows>recent research</a> has shown high accuracy in determining areas of high risk for floods and fires, as well as earthquakes. Another model was developed to predict and mitigate<a href=https://www.nature.com/articles/s41598-020-77757-w> </a><a href=https://www.nature.com/articles/s41598-020-77757-w>hazardous air pollution levels</a>.After a disaster has occurred, AI based image processing algorithms can analyze satellite and drone imagery of the affected areas and determine what is the extent of the damage. Such analysis can make relief logistics much faster and more efficient.NLP based algorithms can monitor social media feeds in real time to get more precise information on the status of certain affected areas. They can also be used to deploy chatbots to make requesting help by those affected faster.<img src=/content/images/image_439849ad8e44cf896e3bb835722f5bf4.jpg><h1>Better work and wealth distribution</h1>While AI algorithms can achieve super-human performance on certain tasks, we know that they cannot completely replace humans. We hope that AI will serve as an enabler, freeing humans from repetitive tasks and enabling us to use our creativity to find solutions to problems we don't yet know how to solve.To achieve this, it is important that the entirety of our workforce, no matter the background, is able to access quality education for low cost.Automated curriculum discovery and tailored content creation tools powered by AI can deliver the next-generation in education technologies. This is particularly important for developing economies where there is the potential to democratize access to education at the level of top-tier universities.Knowledge graph technologies combined with the latest advances in NLP can be used to create a personalized skills breakdown for every person in the workforce. This enables an individual to develop an education plan tailored to their own career goals, as well as match them with open positions that could most benefit from their existing skills.<h1>Ethics and bias</h1>While AI technologies have the potential to revolutionize our society and solve some of the world's biggest problems, they also pose important ethical concerns if deployed without serious consideration.Recently AI algorithms were shown to perpetuate societal bias with respect to minorities. Among other examples, language models can learn biased associations between words describing particular ethnic or religious groups. Generative models learn to generate some face types in preference of others.In light of these examples, it is clear that we cannot rush the deployment of AI technology as soon as we have a working prototype. A more thoughtful approach is required.In "<a href=https://doi.org/10.1038/s41467-020-15871-z>AI for social good</a>" the authors outline the key principles for ethical AI development:<ul><li>Expectations of what is possible with AI need to be well-grounded.</li><li>There is value in simple solutions.</li><li>Applications of AI need to be inclusive and accessible, and reviewed at every stage for ethics and human rights compliance.</li><li>Goals and use cases should be clear and well-defined.</li><li>Deep, long-term partnerships are required to solve large problems successfully.</li><li>Planning needs to align incentives, and factor in the limitations of both communities.</li><li>Establishing and maintaining trust is key to overcoming organisational barriers.</li><li>Options for reducing the development cost of AI solutions should be explored.</li><li>Improving data readiness is key.</li><li>Data must be processed securely, with utmost respect for human rights and privacy.</li></ul>We believe these guidelines form only the minimum set of requirements, and as we learn more surely we will only add to this list. Broadening the diversity of AI developers and including the broader community in AI development is critical to this effort.

The sustainable development goals were set in 2015 by the United Nations to guide our society towards a fairer, healthier and cleaner future.

How AI can help achieve the Sustainable Development Goals

As the world's population grows, we are faced with the delicate balancing act between ensuring a comfortable, healthy and meaningful life for billions of people while preserving the ecological balance of our earth’s ecosystems, already precariously close to destruction. As scientists, engineers, and entrepreneurs, we believe it is our social responsibility to develop technology that can address these challenges.In the last few decades we’ve witnessed an explosive growth in the world’s economic activity, which has lifted millions of people out of poverty. This growth was fueled by rapid scientific and technological progress happening over the last seven decades and the speedy deployment of related innovations. Economic productivity has exploded, but we have paid the price with rising social inequality and environmental destruction. For more balanced growth, we need to innovate to develop business practices consistent with the principles of sustainability, as outlined by the <a href=https://sdgs.un.org/goals>UN’s Sustainable Development Goals</a>.While sustainability has emerged as a key concern for businesses throughout the world due to an increased awareness of the toll that population growth coupled with mass consumption has taken on the environment, many businesses are discovering that sustainable practices also help the bottom line - by cutting costs and increasing sales, their consumers increasingly aware of environmental and societal issues.At Recursive, our goal is to collaborate with large enterprises to create innovative solutions to sustainability challenges, merging in our expertise in AI research and our client's domain knowledge. We’ve identified artificial intelligence as a key technology to not only address some of the symptoms of unsustainable business practices, but also in helping to create completely new innovative approaches that can create a more sustainable economy and lifestyle.We’ve identified four key areas where we believe digital technology and, in particular, AI solutions can make a substantial impact on sustainability: improving efficiency, accelerating innovation, risk mitigation and better work and education. Through the coming year, we will be accelerating our efforts to disseminate information about how to identify what areas in a business can be improved using technology and how to develop such systems robustly.We are particularly invested in making sure AI is used in a fair and balanced way, with the gains stemming from its development being evenly distributed. In recent years, we’ve seen some instances of AI systems being used carelessly and leading to discriminatory decisions with real life consequences. At Recursive we want to make ethics a priority concern during development, by keeping up to date with the latest research on the subject, as well as building a diverse team of experts and advisors to help us see the bigger picture.To summarize, we’ve embarked on this exciting journey at the crossroads of technological innovation and sustainable development because we believe that the timing is right, with the confluence of exciting new developments in the field of artificial intelligence combined with an increasing awareness that we all need to put sustainability first in technological innovation programs.We’re looking forward to many fruitful collaborations with our partners and are hoping to expand our team with talented individuals passionate about sustainability and innovation. If you are interested in any of these themes, please reach out to us for a chat!

As the world's population grows, we are faced with the delicate balancing act between ensuring a comfortable, healthy and meaningful life for billions of people while preserving the ecological balance of our earth’s ecosystems, already precariously close to destruction.

Blog

Recursive's sustainable innovation now and in the future