Wednesday, April 25, 2012

Tested software agent server with Apache Solr trunk (4.x)

Last week I made some enhancements to the new Base Technology Software Agent Server and tested it accessing the recently released Apache Solr 3.6 enterprise search platform. My test was quite simple, one test that added a few documents to a Solr collection, and the other test performing a few queries of that collection, all via the HTTP protocol, using XML to send data and receive results.
 
Earlier this week I downloaded the latest "nightly trunk build" for the next generation of Solr, referred to simply as "Solr trunk" or "4.x". My tests from Solr 3.6 worked fine except for one test case that checked the raw XML text and there one one nuance of difference - in 3.6 a zero-result query generates an XML empty-element tag for the "result" element, but in Solr 4.x a start tag and separate end tag are generated. No big deal.
 
As alluded to last week, I added the option to disable "writing" to the web (HTTP POST.) This option defaults to "disabled", which is safest. You need to set the "implicitly_deny_web_write_access" property to "false" in the agentserver.properties file in order to send documents to Solr from an agent running in the software agent server, but this is not needed if you are simply trying to query an already indexed document collection, which is most of what I was interested in anyway. Having the ability for an agent to actually add documents to Solr was simply an added benefit.

Sunday, April 22, 2012

I'll be talking about the agent server at the next NYC Semantic Web Meetup

I'll be giving a 3-minute elevator pitch for the Base Technology Software Agent Server at the upcoming NYC Semantic Web Meetup, on Thursday, April 26, 2012. That won't be enough time to go into any details, but hopefully will pique a little interest.
 
In preparation, I have refined my short summary as well as a more detailed summary.

Thursday, April 19, 2012

Tested software agent server with Solr 3.6

I just ran a couple of simple tests to see how well the Base Technology software agent server could connect to Apache Solr 3.6 (open source enterprise search platform) which was just released last week. I did have to make a few changes to the agent server code, to add support for the HTTP POST verb and to permit HTTP GET to bypass the web page cache manager of the agent server.
 
Originally, I was going to access Solr via the SolrJ interface (Solr for Java), but I figured I would start with direct HTTP access to see how bad it would be. It wasn't so bad at all. I may still add support for SolrJ, but one downside is that it wouldn't be subject to the same administrative web access controls that normal HTTP access is. I'll have to think about it some more, but I could probably encapsulate the various SolrJ methods as if they were the comparable HTTP access verbs (GET for query, POST for adding documents, etc.) so that the administrative controls would work just as well with SolrJ. At least that's the theory.
 
For now, at least I verified that a software agent can easily add documents to and query a Solr server running Solr 3.6.
 
The code changes are already up on GitHub.
 
I do need to add a new option, "enable_writable_web", which permits agents to do more than just GET from the web. I had held off on implementing POST since it is one thing to permit agents to read from the web, but permitting them to write to the web is a big step that adds some risk for rogue and buggy agents. For example, with one POST command you can delete all documents from a Solr server. Powerful, yes, dangerous, also yes.
 
I also need to make "enable_writable_web" a per-user and even per-agent option so that an agent server administrator can allow only some users or agents to have write access to the web. There will probably be two global settings for the server, one for the default for all users, and one which controls whether any users can ever have write access to the web. The goal is to make the agent server as safe as possible by default, but to allow convenient access when needed and acceptable as well.
 
Unfortunately, after all of that, it turns out that Solr has a "stream.body" feature that allow documents to be added and deleted using an HTTP GET verb. Oh well, that's life. You can't cover all bases all of the time.

Tuesday, April 17, 2012

Scripting Language for the Base Technology Software Agent Server

Much of the processing for software agents in the Base Technology Software Agent Server occurs automatically down inside the server software itself, but occasionally a modest amount of procedural scripting is needed, as well as expressions for numerous parameters. The scripting language for the agent server is based on Java, with a number of simplifications and extensions.
 
Expressions and most statements follow Java fairly closely. Those who are familiar with expressions and statements in C, C++, and Java, should feel right at home. The ++, +=, and ? : operators are supported, for example.
 
Since the purpose of scripts is primarily "small snippets of code", there is no support for defining classes and other complex structures that are supported by Java and C++. On the flip side, since lists and collection of named data are so common, the 'list' and 'map' types are built into the language, and the built-in "web" type dramatically facilitates access to web resources. The expectation is that any complex structures should be constructed by developing collections of agents themselves. Also, simple map objects are very convenient for collecting related information, without any of the tedium of defining and implementing full-blown classes. That said, I am sure that some future stage of the agent server will add support for classes, in some form.
 
The scripting language does not have a 'new' operator, but lists and maps can be trivially constructed without it using list and map literals, more reminiscent of JavaScript than Java. Also, typed fields are automatically initialized to their specified type, so there is no need for 'new' simply to initialize an empty list or map.
 
As far as data types, the scripting language supports most of the Java primitive types, as well as built-in support for strings, lists, maps, and web objects. The various keywords for the types of integers supported by Java are supported, but are all mapped to one long integer type. Similarly, the various keywords for float and double are supported, but are all mapped to double. The 'char' type is supported but mapped to string. String literals may be enclosed in quotes or apostrophes (single quote.)
 
The scripting language also includes support for dates and money, although that support is not yet fully implemented. Strings can simply be mapped between ISO date/time format and RFC date/time format as well as integers.
 
Currently, scripts are simply named functions with no parameters. Syntactically, they are a Java block but without the braces. Nesting of statements and blocks with local variables is supported, much as in Java.
 
Unlike Java, elements of lists and maps map be referenced as if they simple arrays rather than require the explicit 'get' and 'put' method calls (which are still supported for compatibility and due to their familiarity.) Substrings or sub-ranges of strings and lists can easily be extracted using a pair of subscripts to specify the range.
 
Characters and substrings of strings can be accessed directly using square bracket subscripts as well rather than use the 'charAt' method (which is still available for compatibility.)
 
Another simplification is to treat the 'length' and 'size' methods identically regardless of whether the object is a string, list, or map. Further, the useless, empty parentheses can be omitted.
 
As an additional simplification, the value associated with a string map key can be directly accessed with the traditional C/C++/Java dot notation as if it were a class field. This includes both reading and assignment.
 
Quite a few of the Java string operations are supported, in addition to operations such as 'between' to find two substrings and return the text between them. The 'before', 'after' and 'between' operations also have regex-forms to allow very powerful but concise string manipulation, which makes it very easy to extract data from web pages, text files, and XML documents even without using the built-in HTML and XML parsing features. It is also easyt to extract word lists from HTML, XML, text, etc., as well as to generate strings from list and map structures.
 
One significant difference from the Java String class is that substrings can be modified in-place. Individual characters can be replaced, characters can be deleted, inserted, and substrings can be replaced with other substrings whose length may differ. Since the original string is modified, a copy of the string is made whenever it is stored in a variable.
 
Another improvement for strings is that the relational operators can be applied directly to string values, as opposed to resorting to the 'equals' method. The 'equals' method is still supported, as is the 'equalsIgnoreCase' method.
 
The Java library is a bit confusing as to when 'add', 'put', and 'set' are to be used in different classes, so we treat all three identically for list, map, and string.
 
All text is presumed to be UTF-8 encoded UNICODE. Explicit character codes code be embedded within string literals as in Java.
 
The so-called bit-wise operators (&, |, ^, ~) are not supported at this time, but the logical boolean operators (&&, ||, !) are supported. The shifting and bit rotation operators are also not supported at this time, although they may resurface in a future release.
 
One minor nuance is that types are all lower case since they are all built-in primitive types. This includes int, long, string, list, map, and web. There are no "boxed" types as in Java, nor any need for them.
 
Also missing is Java's extensive class library and third-party libraries. But, the built-in 'web' type greatly simplifies access to web resources, including HTML web pages, text files, XML files, RSS feeds, and REST API web services. A very rich set of functions and methods are already built into the scripting language and its runtime, especially with the flexibility of the built-in list and map types. Over time additional types and functions will be added as their need becomes apparent.
 
User-defined functions are supported within an agent. Unlike normal scripts which are Java-style blocks without the enclosing braces, a user-defined function has something very similar to the function header and block block syntax of Java.
 
The developer of an agent may optionally decide that selected scripts and functions are to be "public", which means that they can be called directly using the REST API. This presumes that the user provides the proper user ID and password, so this feature should be reasonably secure. And, of course, the developer can opt to make none of the scripts or functions public for maximum security.

Sunday, April 01, 2012

Agent Server is a product without a name

My new agent server does not have a formal name. I have done this intentionally. I don't actually consider it a true "product" or commercial "service" at this point in time. It has no real packaging (okay, it does have a downloadable zip file and a home on GitHub) and you can't "buy" or "subscribe" to it per se. From my perspective it is still "raw" technology. Yes, it is packaged to make it easy for people to access the technology, but it is certainly not "ready for prime time."
 
I vaguely considered whether to give it a name, but decided that all such "marketing" effort would be a distraction from focusing on getting the technology working and available for evaluation. Any kind of true marketing is still down the road.
 
For now I use one of the following descriptive names to refer to the technology:
  • Agent Server
  • Software Agent Server
  • Base Technology Agent Server
  • Base Technology Software Agent Server
  • Any of the above with a suffix of "Stage 0" (e.g., "Base Technology Agent Server - Stage 0")
Once I start getting some feedback on the technology and refine it a bit more, then and only then will I be ready to engage in some "real" marketing, starting with brand identity, naming, product positioning, messaging, and promotion.

Business model for Agent Server

Although subject to change and evolution, the currently expected business model will be primarily a traditional open source model:
  1. The software, including full source, is "free" and freely available (on GitHub) under the Apache License Version 2.0.
  2. Consulting and contract work to support, extend, and customize the software will be the primary source of revenue.
In addition, I have some preliminary thoughts for a longer-term plan to offer a subscription-based "agent grid" network of web-based servers that are tailored to offering commercial-grade agent server support for organizations that do not wish to host and support the agent server software on their own machines. But, this option is still way off in the hazy future.
 
For now, I seek customer/partners such as large data providers (or any organization who offers a web-based API to their services) who have a strong interest in software agents that facilitate consumption of their data in a way that is compatible with their own business model.
 
-- Jack Krupansky

REST API for interacting with Agent Server

The Base Technology Software Agent Server supports a full-featured REST API for all interactions with developers, users, and administrative control. This API uses the HTTP protocol with its POST, GET, PUT, and DELETE verbs and passes information via the URL path, URL query parameters, and JSON structures. In fact, the complete definition for a software agent can be expressed in a single JSON structure. All information about the server, users, and agents is expressed in JSON, but there is a "format" option to use XML, CSV, or text for various API calls.
 
Administrative controls include starting, stopping, pausing, and resuming the agent server, as well as throttle controls for operations such as web and email access, and the ability to disable and enable individual users and their agents.
 
Individual users can define their own agents, but it is expected that developers will provide the definitions of agents, including their internal scripts, and that users will then instantiate and control those definitions with specific parameter settings.
 
There is one other form of interaction: email notifications. Agents can notify users of conditions, events, options, and choices via email. This could be information-only, optionally require confirmation or a yes/no choice, or even a selection from a list of options. The user may make their selection by clicking a link in the email message, which is actually a REST API call to signal the agent about that choice selection.
 
It is expected and hoped that others will create web and mobile device-based user interfaces. Or, that partners will contract for the development of customized user interfaces for specific applications of software agents.
 
The underlying concept here is that the agent server provides an easy to use REST API that enables the use of software agent technology for a very wide range of applications rather than providing a complete packaged solution for a limited set of applications.

Agent Server is open source Java on GitHub under Apache License Version 2.0

Stage 0 of the Base Technology Agent Server is 100% Java developed using Eclipse and ant, and is licensed as open source under the Apache License Version 2.0. The full source code (and downloadable zip file) is available on GitHub.
 
You may read, download, modify, and even redistribute the agent server and its source code according to the terms of the Apache License Version 2.0, without needing any payments or agreements to be signed.
 
That said, the agent server is not quite ready for prime time, so be prepared for bugs and other issues if you do take an advanced preview. Everything is "as-is."

Components of a software agent

The model of a software agent supported by the Base Technology Agent Server is relatively simple but enables both a sophisticated level of processing and automatic processing by the underlying infrastructure of the agent server. The feature areas or components of an agent in this model are:
  1. Parameters - needed to parameterize the behavior of each agent
  2. Inputs - other agents upon whose outputs this agent depends
  3. Timers - to control periodic behavior of the agent (its "heartbeat", so to speak)
  4. Conditions - expressions which must be true for the agent to take action
  5. Scripts - procedural code to respond to specific events
  6. Memory - internal storage that persists for the life of the agent
  7. Scratchpad - temporary storage that is not guaranteed to persist, such as if a server is rebooted
  8. Outputs - a collection of data fields to be made available to the environment and other agents
  9. Notifications - conditions under which the user will be notified of events.