BLG and AGI Studio

Joel If we want to decide on some kind of standard command line to call the Base Logic Generator from AGI Studio so that the two can communicate, I think that would be cool. I'm thinking something like:

AGIBLG -i

to indicate that the program is being run in an integrated context and that it should behave appropriately (e.g, close automatically when files are generated, not save options such as whether the program should write to a file or to the clipboard, etc.). Of course the integrated behavior would also be customizable. It would be a simple matter to have the program return different values based on whether the user generates a file or cancels the operation.

Any other options that should maybe be included?

Of course, a communication interface would need to be defined. The clipboard is one possibility (the code for writing to the clipboard is already written, after all). Any other ideas?
sonneveld How about pipes you could redirect pipes to and from agi studio and pass along info and commands?

- Nick
Joel what sorts of commands? I'm thinking of the BLG as pretty much a dialog box that comes up when you would choose a menu item to generate a logic template. In which case the only real data exchange needed is to tell the BLG how it should initialize itself and for the BLG to tell AGI Studio (or whatever the calling program may be) whether the user generated the source code or cancelled the operation and if the user generated the source code what the source code is. Are you thinking of something more powerful than that?
Nailhead This is a great idea Joel!

I can't think of what other types of commands that would be useful to pass between the apps, other than what you mentioned, as far as using BLG as a dialog called from AGIStudio.

example:
success=0 or success=1
sonneveld well, sending commands via pipes would be more useful for a sound player (play, pause, stop).. but I thought if we could do some things generically like sending file info.. it could be used in other programs.

- Nick
Joel I don't really know how to redirect pipes (I'm not even 100% sure what it means), but if you have an idea for a standard interface then that's fine with me. I'm simply suggesting that the interface be defined so that I can make the appropriate updates to the BLG. I think it would be good if the version 2.0 full release version already had the capability rather than the ability having to be added in a 2.1 or something like that. On the other hand, I'm not suggesting that we rush the design of the interface, either. I do have a bit of work to do before the BLG is ready for a full release. As you may have noticed, it performs no bounds checking at the moment, and it asks you for the game directory every single time you generate a file (though that problem I've mostly taken care of already).
Mokalus_of_Borg Well, then, like good little software engineers, let's start with...
[size=3]Requirements[/size]
  • What do we want to achieve here?
  • What information do we need to send between the two programs to achieve it?
    Mokalus of Borg

    PS - It'll make the rest of it easier.
    PPS - Suggestions? Comments? Airborne rotten fruit?
    • Joel I'm not totally sure what we're trying to achieve here. As far as the BLG is concerned, all that needs to be achieved is that AGI Studio needs to be able to say, "BLG, start up and let the user generate a logic, then tell me whether the user actually generated a logic, and if they did then tell me what the logic was," and the BLG should respond appropriately.

    However, it would be more versatile to design an interface that allowed any AGI programs that implemented the interface to communicate whatever types of data they're capable of communicating.

    As I see it there are five types of resources:

    - views
    - pictures
    - sounds
    - compiled logics
    - logic source code

    The BLG is capable of communicating the last 2 but in the context of AGI Studio I think it would be more likely communicating just the logic source code.
    sonneveld Yeh, sorry... I didn't want to get things too complicated that you wouldn't bother to get any sort of communication between agistudio and blg. It was just an idea.

    Another application for better communications between apps would be for a debug mode in NAGI. setting breakpoints in agistudio and then running nagi until it reaches one would be terrific.

    so really you'd want some sort of protocol to send file information, breakpoints, settings or commands.

    - Nick
    Joel that debug mode thing with NAGI does sound cool.

    to reduce the complication of this thing, how about designing communication protocols in terms of application classes?

    for example, a logic template generator would be one application class needing only a logic template generator communication protocol. The BLG would need only implement that simple protocol to be able to communicate with AGI Studio (or more generically, the IDE application class).

    there could be an interpreter application class which needs to obey an interpreter communication protocol in order to communicate with AGI Studio.

    of course, going this route there are things to look out for, such as designing the protocols specifically for the tools that exist at this very moment and not in a generic way that would allow extensions. and, of course, it would be advisable that any tools that are developed continue to produce resources that can be imported into AGI Studio even if AGI Studio can't just get the thing by "simple" communication.
    Joel another possibility -- do you think sockets would be a good communication platform? the only Win32 stuff I can find regarding pipes talks about creating named pipes, but only WinNT/2000/XP can create them. Windows 95, 98, and Me can't.
    sonneveld I was thinking more of redirecting the stdin and stdout for communications. I believe they're just unnamed pipes.

    Sockets was my other choice.. but I wanted it to be simple to implement for all apps.

    Microsoft has a nice article on all the interprocess communications they have. If we want to have source that's compatible with unix.. pipes and sockets are probably the only ways to go.

    - Nick
    Joel Ok, I got a program to work with redirecting unnamed pipes. This may be simply because I just finished a network programming course and this is the first time I've used pipes, but it seems to me that sockets are more straightforward, at least for the parent process, which really seems to have to jump through hoops to set up all the pipe redirection.

    The only problem I see with using sockets is that Windows doesn't seem to support the PF_UNIX (a.k.a. PF_LOCAL) style communication (although I may just be doing it wrong -- I can't find anything about it on MSDN), but that could possibly be overcome using PF_INET instead and specifying the server address as INADDR_LOOPBACK. It depends on how portable that solution is, but that would allow the use of TCP or UDP on the local machine without needing to know the local machine's IP address. I remember reading about some kind of issue with IPv6, but I assume there will continue to be a loopback address in IP version 6, so there may just need to be a recompilation if there's a problem.

    On the other hand, pipe redirection is fairly simple on the child end of things, so I guess as long as pipes are as versatile as sockets in a local machine context, then I guess I don't have any problem with using them.
    Nailhead The problem with sockets is it would ring people's firewall software and therefore worry them about what's really going on with the app.

    What about using a DDE connection?
    Joel I thought of that problem with firewall software, actually. But anyone who's really, really concerned about it to the point where they might consider blocking the communication would hopefully recognize 127.0.0.1 or whatever as the loopback address. Of course, using any of our fan-made AGI software also requires a bit of faith in the authors not to be doing stuff they shouldn't be doing. I agree that it is an issue, though.

    The problem with DDE is I think that it's Windows-specific, and it requires broadcasting messages to all the programs running in Windows to get the communication started. I think we should design the communication protocol so that Linux, etc. users aren't left out of the possibility of utility integration.

    I think Nick's got it right. If we want to be portable, we're probably going to have to use either pipes or sockets. Where each of them stands is pretty much like this:

    where "server side" refers to the utility application such as the BLG and "client side" refers to the application that invokes the utility application, such as AGI Studio

    sockets

    the good:
    - slightly less complicated than pipes on the client side; possibly less work for Nailhead :)
    - Internet capable; whatever possible application that might have I don't know
    - documentation is easy to find
    - highly portable...the Windows Sockets API is designed to be compatible with the Berkeley Sockets API

    the bad:
    - slightly more complicated than pipes on the server side; may prevent novice programmers from developing integration capable AGI utilities
    - could trigger firewalls; might make some people suspicious...might be a security hole

    Pipes:

    the good:
    - very easy to use on the server side
    - no triggering of firewalls

    the bad:
    - not so easy on the client side; I got it to work but I only understand what it's doing in an intuitive way; the MSDN example code is not completely clear
    - no Internet capability - in order to be compatible with Windows 95, 98 and Me, we have to use unnamed pipes, which can't be created across a network connection
    - documentation is not as easy to find
    - less portable when doing things the MSDN example code way -- althought Unix may support pipes, it wouldn't set them up the same way the MSDN example code did
    sonneveld with firewalls.. in relation to security:

    on unix boxes.. I'm pretty sure they use loopbacks all the time. so no firewall would block out 127.0.0.1. it should block out anybody trying to send 127.0.0.1 across the network though.

    If a windows firewall tries to block it or shows a warning then it's just plain broken as far as I'm concerned.

    - Nick
    Joel ZoneAlarm for Windows does give a warning for IP address 127.0.0.1. I think, however, that we could just make note of the fact that 127.0.0.1 is the loopback IP address and if anyone wanted to check it out then maybe provide a URL where they can verify. Basically, there's no more risk that an AGI utility will communicate data over the Internet that you don't want communicated than there is that an "AGI utility" will delete all the files on your hard disk. Besides, this is one of the reasons why I suggested that no utilities be made that require communication with AGI Studio. If someone's paranoid about it, they can just not use the utilities in an integrated context (barring ones where integration is absolutely necessary such as debugging with NAGI) and instead write the resources to disk and then import them into AGI Studio or write the logic source to the clipboard and then paste it into AGI Studio, etc.
    Joel RFC1812 should be an official-enough document to refer people to. It's the RFC for IP version 4 router requirements, which states that any packets with a destination on network 127 should not be forwarded except over a loopback interface or something like that. Plus, a general search for loopback address on the Internet provides plenty of information.
    Nailhead While balancing the good and the bad of both protocols, I'm going to vote for using the localhost address using sockets. The big reason for me leaning towards this is the fact that I'm very familiar with sockets and have dealt with using them in the majority of applications that I write. I realize that this may thwart new AGI Utility authors but I think sockets may be more known than pipes (I could be wrong).
    sonneveld so what would the protocol be?

    agi studio executes a nagi or blg like this:

    "program -i 23456" where 23456 is the port on the local computer?

    - Nick
    Nailhead Yeah, that's a start. The port number is an important thing to pass to initiate the communication. Is there anything else that needs to be passed? I can't think of anything.
    Joel Honestly, I think the biggest difficulty in setting up socket communication for an integration-capable AGI utility would be getting the server started, and info on how to do that would be fairly easy to find, I think. Otherwise, I don't think pipes are much different than a TCP byte stream.

    I don't think any other information needs to be passed on the command line in order to set up the socket communication, although we may want to have some method of error recovery in case the port given on the command line cannot be binded (bound? well, bind is the function name, so I'm going to say binded).

    As for the protocol after the communication is set up, that may need to be designed on an application-class basis. The logic source generators should probably have some method of specifying where in a file that code they generate is supposed to go. For the current implementation of the BLG, the code would always go at the beginning of the logic and replace everything that's already there, if anything is there. For something like blacki's generator, the code might be inserted in the new room section or in the main part of the logic after the new room section or at some arbitrary point (such as inside the code block for if said("hi", "there")).

    As I said before, the trick here is going to be getting away from the existing applications and designing for more abstract applications. And we might need to take into account the possibility that a program is a hybrid of two different application classes (and for that reason the application class way of looking at things may not work, but then again it might). The BLG is technically a logic source generator and a logic compiler, but it's a compiler that can't take logic source code as input.
    sonneveld If we're talking about the server. I thought AGI Studio would be the main server since that would be the program open all the time.

    Even NAGI, with a debug mode, could be a client to the studio. It would simplify code for the client.. all it would have to do is open a socket and give an error if it can't connect (through no ports available.. or the server isn't replying).

    with debugging.. I've been thinking about breakpoints and defining them.. If we want to define breakpoints on a line basis.. we'd need some sort of map, associating lines with offsets within the logic code. The only thing that could do that would be the compiler.. since NAGI wouldn't know (it could guess.. but it would wildly off depending on how the source was organised).

    - Nick
    Joel Actually, AGI Studio is really the client program, since a client program is a program that requests another program to do something on the client's behalf. I think it would make more sense in most cases if AGI Studio requested, for example, the BLG to generate a new logic for it than for the BLG to request that AGI Studio accept a logic that was just generated. I say that only because I think the average user will probably open AGI Studio and want to start utilities from there as they work. They will probably not want to start the utilities and then start AGI Studio and expect AGI Studio to accept things.

    On the other hand, there is the possibility that NAGI would be running and the user could request that the debugger be started. In that case, AGI Studio does become the server and NAGI the client. There could be a mechanism for starting AGI Studio as a server that will then become a client. As in:

    1. User runs NAGI
    2. User requests that the debugger be started for the currently running program
    3. NAGI starts AGI Studio - at this point NAGI is the client and AGI Studio is the server
    4. NAGI indicates to AGI Studio the folder where the currently running game resides (this would be problematic, however, because I don't believe AGI Studio can open the game while it is running)
    5. NAGI indicates that it wants AGI Studio to act as a debugger
    6. At this point, the roles reverse -- AGI Studio becomes the client and NAGI becomes the server
    7. AGI Studio interacts with NAGI the same way that it would if NAGI had been invoked by AGI Studio

    I would definitely say it would be the compiler's responsibility to maintain a map between the source code and the compiled code. Basically, the compiler would determine, based on where the breakpoint is in the source, which instruction in the compiled logic is at the breakpoint. It would then need only report the compiled code offset for the breakpoint to NAGI. When NAGI hit the breakpoint, it would report the offset into the compiled code, which the compiler would then map back to the line number. There really isn't any need for for NAGI to even know the line number in the source code except maybe for printing out error messages. But even the error message responsibility could be left to the compiler.
    sonneveld You're right when it comes to the actual definition of client and server. I just meant that when it came to initiating communications, if you have agi studio always waiting for programs to communicate to, you just need the client to say hello. once you have a proper link, they can decide who they want to be.

    um.. I hope that makes sense. :)

    the utility, when it talks to agi studio, it could say,

    (data generator sending multiple files)
    "hello. I'm a data generator. waiting for user input.. please wait"
    studio: "ok. I was expecting you to call"
    10 minutes later
    "I've written this logic text.. I would like it to be logic 10 in the currently opened game. I also have other logics 3 and 4"
    <file 1 DATA>
    <file 2 DATA>
    <file 3 DATA>
    studio: "ok, saving it.. done. you can quit."

    (or with NAGI. game doesn't start until studio says so)
    "hello. I'm an interpreter. waiting for commands"
    studio: "I'd like breakpoints here. "
    studio: "start your game"
    <game plays>
    "reached breakpoint. game paused"
    studio: "user has looked at it.. changed code."
    studio: "restart game"

    (or with something else)
    "hello. I'm a data generator."
    studio: "sorry. I'm looking for a debug interpreter"
    "oh.. sorry"

    that way you'd have some sort of check to make sure that you have the right kind of client running. You would have to define certain kind of apps like generator, and debug interpreter.

    one problem would be if agi studio quit or swapped games WHILE the helper utility is still running. You would need some other communications telling the status of agi studio and confirming it with the utils.

    one solution is for agi studio to go into some sort of "waiting for client app to finish" but what if something goes wrong? you'd need to cancel and a way to let the client app know that it's running out of context (if it's still running later on)

    with files not being openable in agi studio when the interpreter is running. I think it may have something to do with agi studio opening files to write them. perhaps if it just opened files as readonly it wouldn't conflict with nagi. only when it needs to write would it have to close and reopen them.

    - Nick

    PS.. I think I rambled a bit.
    Joel I don't think it would be a bad idea for AGI Studio to listen for connections from the utilities so that if they are started separately they can still communicate; however, I'm not sure it would be easier to do things like that in the general case. In general, I think it would be easier for the utility to be pretty much a dumb server. It assumes that any app. that runs it in an integrated context is an app. that can handle its output. If it doesn't understand things the client is sending to it, it just ignores them. We could design a basic protocol where the utility is a dumb server only and an extended protocol for the utility to be able to connect itself to an app. that can handle its output. Then, utilities can implement the basic protocol always and the extended protocol only if the author really wants to. The big issue with that is how the utility knows which port to connect to when trying to talk to AGI Studio. When the utility is a dumb server, it's given a command line argument. There's no such mechanism if the utility is responsible for connecting itself to AGI Studio.

    As for the user quitting AGI Studio while the utility is running (or the other way around)...when using TCP, if a recv call ever returns 0, then I'm pretty sure it means the connection was terminated some way or another. As long as that condition is tested for, I guess the individual apps can determine how they react to the connection going down.
    Joel To clarify, I guess that an app. such as AGI Studio could close the connection to a utility process when the context in which the connection was started is no longer valid.

    I just realized a minute ago exactly what situation you were referring to. I'm not sure what the best way to handle that would be.
    sonneveld with ports.. you wouldn't have to worry about them as much is AGI Studio was the server.

    If it was the server it would have a certain port open all the time. There could be a default one or you could define another one if there were conflicts.

    ie.. client checks port 5000 for AGI studio by default
    or if you call client like this: client -p 400.. it checks port 400 instead.

    And yes.. I agree with you about utils not know all the different protocols. There could be a base protocol for sending the status of agistudio and the type of utility. then you could define a debug protocol on top of that that only nagi would need to know. Or a file generator protocol for logic generators.

    only agistudio would need to know all the protocols because it would be communicating with lots of different types of programs. That would make agi studio ideal to be the "server"

    --

    re: which one is server.

    it depends on which program is going to be loaded up first. If agi studio is loaded first.. it would be useful for it to be server.. because all the other utils would have an open port to send data to already. if the other app is opened first.. it would have to call agi studio first.. wait for it to open a port and then send data.

    but how would it know when agi studio is ready to receive data? "ping" it (send a hello msg)? check when that port is opened? in that instance it would be useful for the util to be the server instead because the util could open a port and then agi studio would send stuff to it. you couldn't have a "standard" port though.. each util would have a different port number.

    but you can't really have one be the server sometimes and then the other be the server in other instances..

    --

    this is all theory though.. and you can say that I'm talking out of my ass. I'll try and knock up a prototype and see which is more practical.

    - Nick