Using AI, or Large Language Models (LLMs), is a powerful way to interact with data. For a user, using normal language to interact with underlying systems and information to retrieve processed and evaluated details is intuitive and powerful. The ability to converse with an underlying system iteratively, where the AI agent makes queries and evaluates the information, can greatly simplify the retrieval of information.
The addition of AI chat technologies to Mesh Scope has created new opportunities to understand the mesh network and node activity. Although this is a new addition to the system, it already has created interesting query models that were previously difficult to perform.
AI Augmented IoT Mesh Data
The foundation of the Mesh Scope AI chat capability is the enriched data created during the data processing and storage pipeline, described in other recent Lemuridae Labs blog posts. The Lemuridae Labs team then this foundation and created a new query capability to use natural text to interact with the system.
The AI integration within Mesh Scope is built around the concept of “function calling” or “tools”, where the LLM is provided the ability to query the system data in flexible and far-reaching ways. We created various query and data summary interfaces and incorporated these into the AI framework in support of the chat functions.
The query model produced is focused on node metadata and position details, enriched within the data pipeline processing, and is supported by other geospatial query and aggregation functions. Although there are many ways that this capability could manifest, the current implementation is extremely flexible and provides interesting results with minimum effort for the user.
Another aspect of AI technology is called Retrieval Augmented Generation (RAG), where the AI model has processed content into embeddings and represented data in a vector space, to be used synthesizing new information, summaries, and responses. Although this is a powerful technology, it works best where a large amount of textual context exists to support the process. In this case, the data is structured, not free text, and is best served through the structured query process, and then allowing the LLM to synthesize the results for the chat.
AI Disclaimer
The LLMs that drive many of the modern AI functions are not perfect and ultimately do not truly understand the information being processed and presented. This can lead to odd responses at times, and sometimes rephrasing the question can lead to better results. For example, asking for nodes active around a specific city might search the city only, or might search a radius around it.
Essentially, you can ask an AI for a hammer and most of the time it will give you the hammer, but on occasion it will return a screwdriver and start poking at the nail. Being clear in the direction helps the LLM to better understand the indent and potentially produce better results, and finding edge cases that fail to tweak and adjust the system over time is key when building AI augmented software.
Basic AI Queries
The first set of chat messages below show the ability of the system to provide direct clear summaries of information. These queries are relatively direct matches of node data, and do not require and contextual information or other Meshtastic metadata to execute. The AI model has the context on the data stored within Mesh Scope, and has been provided directions on how to evaluate the questions provided by the user.
The screen below shows a simple query and response process, which is basically a direct query of the underlying data within Mesh Scope.
As can be seen, the LLM provides its own summary and point of view on the information retrieved, and has chosen to detail some aspects of the device configuration. Given that there is not a direct requirement on presenting the node information, this interpretation works well, and makes the experience more varied and dynamic.
Advanced AI Queries
More advanced queries start to leverage the knowledge base within the AI models to augment and support the query. Using more indirect matches such as the term “Europe” as opposed to a specific country, allows the AI model to generate much richer queries. This augmentation greatly simplifies information retrieval, although at some risk to LLM interpretation.
As noted above, the more clear and concise a query is, the better the AI model can interpret it as desired, while more vague terms will cause the system to make educated guesses as to the intent of the query. This may lead to unexpected results, or the need to send multiple queries to refine the process.
The query below relies on the LLM to provide the country details when running the data queries within Mesh Scope, while the second builds on this to try to identify out of region nodes.
The ability of the system to perform geospatial queries builds on the underlying data received and creates interesting opportunities. The query shown below makes it easy to do a bounded range geospatial query and to interact with the results. Normally this process would be feasible but potentially challenging, and many novice users would not have the ability. In this case, simply asking for the desired range returns the matching results.
The AI augmented queries can be even less precise, although this may lead to unexpected results. The ability to specify a more general region can be effective, although care must be taken when allowing an LLM to interpret directions. A generalized detail such as “north east Australia” may have different meanings for different people, so precision should be considered.
That said, the query below returned reasonable matches and useful context to the matched nodes.
Interactive Chats
As the chat session has a “memory”, followup questions can be asked from the earlier responses. This interactive cycle is powerful, allowing a user to delve more deeply into the Mesh Scope data. The AI model augmentation and processing on the information retrieved opens up many new possibilities.
The sequence shown below builds on the results from the prior screen shown, and allows the user to simply dive into the results in a simple and conversationally intuitive manner. The second question is an interesting one in that it asks the AI model to not simply retrieve information but to evaluate it and return the details. Again, care should be taken when reviewing AI generated results, but the process is a powerful one and can be quite successful.
Indirect or Relative Queries
Given that the AI model is capable of performing multiple queries to support the task, new indirect queries can be given. The example shown below is very interesting, as the query does not indicate anything about where a node might be, simply identifying the desired mesh node.
In this simple query, there are multiple data sets retrieved and then processed to identify distance and even bearing, which was all summarized in the node details. Generally this would take a fair amount of effort to perform, and yet in this case the AI model completed the task easily.
Localized Chat Results
The last area to review is the ability for the AI model to automatically localize information on behalf of the user. The Meshtastic network is global in nature, and there should be no expectation that users will all interact in English. Many language models are multi-lingual by nature, and this provides the ability for users to interact with data more naturally, letting the language model handle the translation.
Obviously machine translation may not be perfect, but as the screen shot below shows, it can be effective and immediate.
Beyond the direct language capabilities, the chat can automatically interact with a user in the languag of their choice, within reasonable bounds. The final example, shown below, is a user asking a question in Portuguese. The system automatically processes the question, evaluates the results, and returns them in the proper language to match the request. Again, as AI is not perfect some confusion may occur, however this is a positive step towards supporting multi-lingual environments and teams.
Summary
Lemuridae Labs works with complex information systems and seeks to find better ways to process, evaluate, and interact with this information. A chat interface, although not perfect, can be a powerful tool for an individual quickly seeking answers from a system. Although certainly a user could export data and run a series of queries and other processes on the informaion, that requires specific skills and capabilities, not to mention time. Being able to streamline the process and answer the question, while avoiding complex and potentially unnecessary processes, is beneficial.
That said, AI systems are not magic, even if they appear to be at times. Producing an effective solution require planning, iterative testing, and refinement. The team working on the effort must set goals and milestones, and understand the capabilities and limitations of the technologies being employed. Only through a comprehensive process can the desired results be achieved with the quality and effectiveness required.
We are always happy to discuss our capabilities, build partnerships, and otherwise collaborate. Feel free to get in touch!