the Freiburg project FAQ

Table Of Contents

• 1. How to make different services communicate?
• 2. What exactly does 'libfreiburg'?
• 3. How does one service do to understand the data another service is sending to it?
• 4. It is likely that Freiburg will similarly need a common basic format?
• 5. What does Freiburg not do?
• 6. What is the overhead of Freiburg?
• 7. It's not clear to me how many instances of a service might exist
• 8. It's not clear where the state for a service resides
  • 8.a When Freiburg was first being mentioned, it was described as a replacement for shared libraries.
  • 8.b My expectations were conditioned by how I understand shared libraries operate. It is now more clear that Freiburg is an alternative to shared libraries that has characteristics much different from them.
• 9. It's not clear how services are shared
  • 9.a One interpretation is that multiple clients connect to the same server. The server might have a common resource (e.g. a database) that the clients want to share. How does one go ahead with that?
  • 9.b Does the above mean that the communication between e.g. ArsService and ArsServer is something that Freiburg cannot handle, because that communication can be many clients to one server?
  • 9.c what is the difference between Ars - API and Freiburg - API?
  • 9.d does this mean I always have to provide an independent process like ArsServiceX to use an already available application?
• 10. Are services supposed to be reentrant or serially reusable
  • 10.a Can an already started server be reused for new requests?
• 11. Do services have to be locked so that they can complete processing complete requests
  • 11.a How does a 'service' detect different 'context' requests''
• 12. How do they signal success or failure

1. How to make different services communicate?

• every libFreiburg 2.0 communication link is a link between one client and one server
• the link is created and maintained by libFreiburg 2.0 as a socket communication using unix-domain-sockets or inet-domain-sockets
• client and server are using the same communication library called libfreiburg

2. What exactly does 'libfreiburg'?

• find and start the service (e.g. server)
• setup and maintain the socket communication interface between client and server
• choose the right byte-order (little/big endian)
• verify 'service' api version
• if an error happen, propagate the error to the client
• do all the send and recv stuff
• do (if necessary) the casting of different types
• provide an event-loop to smoothly integrate multiple tasks
• after the work was done do a clean shutdown of the communication

3. How does one service do to understand the data another service is sending to it?

• DEFINITION: understand means knowing the exact definition of the procedure called.
• client and server using the same protocol as part of libFreiburg 2.0
• the protocol layer is totally hidden to the programmer
• the programmer (e.g., server and client) using the same interface
• the interface is the name of the service provided (e.g., ECO2), the arguments expected (e.g., MQ_INT2) and the return value (e.g., MQ_INT2)
• the 'interface' version is the 'service' api version

4. It is likely that Freiburg will similarly need a common basic format?

• [PACKAGE]=[HEADER]+[BODY]
  • [HEADER]

    Segment	Size	Type	Default	Description
    HDR	3	string	"HDR"	fixed prefix
    Separator	1	char	'+'	fixed separator
    BinaryFlag	1	char	'S' or 'B'	string or binary mode
    StatisticFlag	1	char	'1' - '9'	statistic level
    DebuggingFlag	1	char	'1' - '9'	debugging level
    VerboseFlag	1	char	'Y' or 'N'	using verbose output?
    Separator	1	char	'+'	fixed separator
    MyContext	10	pointer	0x????????	my MqS object
    Separator	1	char	'+'	fixed separator
    RemoteContext	10	pointer	0x????????	remote MqS object
    Separator	1	char	'+'	fixed separator
    BodySize	10	integer	0123456789	the size of [BODY]
    Separator	1	char	'+'	fixed separator
    Token	4	string	"????"	handle name
    Separator	1	char	'+'	fixed separator
    TransactionPtr	10	pointer	0x????????	MqTransS object
    Separator	1	char	'+'	fixed separator

  • [BODY]=[PREFIX]+[ITEM]+[ITEM]+...
    • [PREFIX]

      Segment	Size	Type	Default	Description
      BDY	3	string	"BDY"	fixed prefix
      Separator	1	char	'+'	fixed separator

    • [ITEM]...

      Segment	Size	Type	Default	Description
      Size	6	integer	0123456789	the size of Value
      Separator	1	char	'+'	fixed separator
      Type	1	char	MqTypeE	the type of Value
      Separator	1	char	'+'	fixed separator
      Value	?	binary	??????????	the [ITEM] data
      Separator	1	char	'+'	fixed separator

• reading and writing the format is done by the libfreiburg library

5. What does Freiburg not do?

• no login verification
• no encoding or decoding of binary data
• no encryption or decryption of binary data

6. What is the overhead of Freiburg?

• Win32 dll stripped currently has 76288 bytes.
• Linux on Athlon with 1040MHz using Unix Domain Sockets has a transaction overhead of 20 usec (this is the time for a round-trip: calling remote procedure; remote echo of the argument and wait on result).

7. It's not clear to me how many instances of a service might exist

• INTRO: client/server versa parent/child
• every client-parent-context start its own server (e.g., service)
• if you using the remote -inetd feature every connection to the inetd server starts a single service.
  (e.g. 100 user + 1 client-parent-context per user -> 100 service processes)
• every started service is totally independent from the other services might exist might exist
  • easy load balancing (e.g using multiple hosts as inetd service providers)
  • easy to use multi-processor hosts

8. It's not clear where the state for a service resides

• DEFINITION: The "state" ought to be the totality of everything that can make two identical sequences of libFreiburg 2.0 library commands return different results. A typical case could be if one wants to wrap a Tcl shell up as a libFreiburg 2.0 service; this then has an internal state (values of variables, commands defined, etc.), and it would have to be part of the total state of the service. Two subsequent invocations of the libFreiburg 2.0 equivalent of [[send $service {incr someVar}]] should return different values.
• INTRO: client/server versa parent/child
• my solution: the libFreiburg 2.0 - tcl - service is a link between

   libtcl*.so libfreiburg*.so main.c context.c
  

• to say it in clear words: libFreiburg 2.0 - tcl - service is not the same as tclFreiburg 2.0
  • libFreiburg 2.0 - tcl - service is a server providing a tcl interpreter as a service
  • tclFreiburg 2.0 is a client using the libFreiburg 2.0 - API to connect to any libFreiburg 2.0 - service
• main.c does the setup and starting the libFreiburg 2.0 event-loop

   ErrorCheck(MqProcessEvent(msgque, MQ_TIMEOUT, MQ_WAIT|MQ_FOREVER));
  

• context.c implement one single service called EVAL using one STRING value as input and one STRING value as return
• the parent-context is just the initial tcl interpreter
• the child-context is just a child tcl interpreter

  typedef struct ContextS {
     struct MqS        *msgque  ;  // this is the Freiburg handle per context
     struct Tcl_Interp *interp  ;  // this is the tcl interpreter per context
  } ContextS;
  

• the service-handle in C look like:

  static ErrorE EVAL(MsgqueS *msgque, void* data)
  {
     ErrorCheck(MqReadC(msgque->read, &buf));   // read string
     ...
     if (Tcl_Eval(msgque->myCtx->interp,buf) == TCL_ERROR) goto error;   // do the 'eval'
     ....
     MqSendSTART(msgque->send);
     MqSendC(msgque->send, Tcl_GetStringResult(msgque->myCtx->interp));   // the result back
     MqSend_RET_OK(msgque->send);  // everything is OK
     return MqSendEND_RETR(msgque->send,NULL);  // do the send
  
   error:
     // ... set the error
     return MqSendRETURN(msgque->send, ERROR_PROC_FMT, __func__);
  }
  

• every client-parent-context get its own tcl interpreter that's it

8.a When Freiburg was first being mentioned, it was described as a replacement for shared libraries.

• from the client point of view it is an replacement
• OLD: I have to link with libXXX.so
• NEW: I have to link with libfreiburg.so and using service XXX
• NEW: no need for libXXX.so, no linking, no searching for the right library for my OS, no errors on linking again a wrong version of library, no need for installing all libraries on my host and last but not least your tcl event-loop will always been working

8.b My expectations were conditioned by how I understand shared libraries operate. It is now more clear that Freiburg is an alternative to shared libraries that has characteristics much different from them.

• I personal call it a network shared library or just a service
• from a programming point of view writing applications based on services is much more easy as writing applications based on shared libraries
• reuse service based code is a lot easier as reusing shared library based code
  (Example: libXXX.so need a libperlXXX.so, libtclXXX.so, libphpXXX.so, libjavaXXX.so. libc#XXX.so, infinite long ... interface library but only one libfreiburg.so interface library)

9. It's not clear how services are shared

• DEFINITION: shared as think of it means that the same code can be used by multiple threads of execution simultaneously.
• a service usually has: libSomthing.(a|lib|so|dll) + libFreiburg.(a|lib|so|dll) + main.c + context.c + ... *.c
• sharing code means using dynamic linking (so|dll) rather than static linking (a|lib)
• sharing a running service is available as:
  • only one client connect to one server
  • every client has one parent-context but multiple child-context
  • an advanced event-queue is available to serve multiple requests at the same time

9.a One interpretation is that multiple clients connect to the same server. The server might have a common resource (e.g. a database) that the clients want to share. How does one go ahead with that?

• INTRO: client/server versa parent/child
• DEFINITION server:
  • a server is an application which can handle multiple client connections and serve some kind of services, with the help of a server-API, to the client (example: Sybase/Oracle/Db2/... database, Ars, SAP, PeopleSoft, ... infinite long ...)
  • a libFreiburg 2.0 - server is a wrapper around the server-API form 1. to disconnect the client from the server-API
• in libFreiburg 2.0: one libFreiburg 2.0 - server supports one client but every client supports multiple libFreiburg 2.0 - server.
  • libFreiburg 2.0 was designed to replace server specific API's (as static or dynamic libraries)
  • libFreiburg 2.0 was designed to be very simple and easy to use for programming purpose
  • libFreiburg 2.0 was designed to solve the every 'scripting' language maintain its own link between application API's and language API's problem
• example: Ars is an application server using a [SQL] database and serving work-flow like applications. Ars has its own user management and providing it's own API for login, query, modify ... . the ArsService created by libFreiburg 2.0 does not replace the Ars server. a libFreiburg 2.0 ArsService is used to replace the Ars specific protocol and the Ars specific API with a common usable API. in this case you can call it a protocol-converter with additional features to pre-process the data.

  OLD:               Database       NEW:       Database                                      *
                         |                         |                                         *
                     ArsServer                 ArsServer                                     *
                     |       |                 |       |           <- Ars-Protocol           *
  Ars-Protocol ->    |       |           ArsService1  ArsService2  <- Freiburg-API + Ars-API *
                     |       |                 |       |           <- Freiburg-Protocol      *
  Ars-API      -> Client1 Client2           Client1 Client2        <- Freiburg-API           *
  

• OLD: your client depend on Ars - Api
• NEW: your client depend on libFreiburg 2.0 - Api
• OLD: if you keep in mind that every application (e.g. Ars, SAP, Peoplesoft, Oracle, Sybase, MqSeries, ... infinite long) providing its own API you will recognize that your application(s) will depend on all these API's
• NEW: it doesn't matter which application you are using, there is only ONE API -> libFreiburg 2.0
• to use a new, already available, libFreiburg 2.0 - service in TCL you only have to write a wrapper in TCL and NOT in C using the, already available, tclFreiburg 2.0 API

9.b Does the above mean that the communication between e.g. ArsService and ArsServer is something that Freiburg cannot handle, because that communication can be many clients to one server?

• just to be clear I speak about an already working project, not any kind of want to be
• if you look to NEW (9.a) you see that many clients (e.g. Client1, Client2, ...) connect to one server (e.g. ArsServer) using many services (e.g. ArsService1, ArsService2, ...).
• the difference between OLD and NEW is that:
  • OLD: ClientX is using the Ars - API direct
  • NEW: ClientX is using the libFreiburg 2.0 - API and ArsServiceX is using the libFreiburg 2.0 - API plus the Ars - API

9.c what is the difference between Ars - API and Freiburg - API?

• Ars - API: handle the communication between ArsServiceX and ArsServer using the Ars - Protocol (every application-API providing methods like init (ARInitialization), login (ARVerifyUser), send something (ARSetEntry,...) and read something (ARGetEntry,...))
• Ars - API: is part of the Ars distribution of Remedy an is a closed source library
• libFreiburg 2.0 - API: handle the communication between ClientX and ArsServiceX using the libFreiburg 2.0 - Protocol

9.d does this mean I always have to provide an independent process like ArsServiceX to use an already available application?

• YES and NO
• YES: if the application is 'closed' source (e.g. commercial) you have no chance to integrate the libFreiburg 2.0 - API into the application
• NO: if the application is 'open' source you can integrate the libFreiburg 2.0 - API direct into the application:

                 YES:    'closed'          NO:     'open'                      *
                        Application              Application                   *
  Application-API: ->   |         |              |         |                   *
                     Service1 Service2           |         |   <- Freiburg-API *
  Freiburg-API:    ->   |         |              |         |                   *
                     Client1   Client2        Client1   Client2                *
  

• NO: libFreiburg 2.0 is able to use an external created file-descriptor as socket communication. the inetd feature already using it (e.g. the inetd daemon doing the waiting on incoming connections (e.g. accept) and starting the according service from /etc/inetd.conf with files-descriptor 0 as external created file-descriptor)

10. Are services supposed to be reentrant or serially reusable

• INTRO: client/server versa parent/child
• DEFINITION: Reentrant means that the same body of code can be used by multiple threads of control at once, provided that each thread supplies its own data segment for the code to use.
• libFreiburg 2.0 does not use threads (e.q. I program libFreiburg 2.0 to be thread ready but currently I don't use it)
• threads are only useful for the multiple clients to one server case which is currently not supported
• one client to one server case: the operating system is doing the scheduling of the different server-service-process's
• DEFINITION: Serially reusable means that the same body of code can be used by multiple threads of control, provided that execution of the code is allowed to complete before another thread can use the shared body of code.
• libFreiburg 2.0 is serially reusable based on a very advanced event-loop (like tcl)
• every transaction send $CTX -wait .... is calling the event-loop to process additional events, even of the same type

10.a Can an already started server be reused for new requests?

• INTRO: client/server versa parent/child
• one parent-context and multiple child-context per server are supported
• every server-context (parent or child) can create additional client-context
• every client-parent-context start and stop a new server
• every child-context is reusing the server-service
  using 'Tcl' code

  set Ctx [ArsService::Start -name MyApplication -uds];	  # client-parent create and start >>ArsService<<
  ArsService::Login $Ctx -wait LOGI User1 Pwd1 Server1 Enc1;# client-parent login into server1
  ArsService::DoSomeWork $Ctx -wait DOSW ....               # client-parent is doing some work on server1
  set Cx2 [ArsService::Start -parent $Ctx]                  # client-parent create a child
  ArsService::Login $Cx2 -wait LOGI User2 Pwd2 Server2 Enc2;# client-child login into server2
  ArsService::DoSomeWork $Cx2 -wait DOSW ....               # client-child is doing some work on server2
  ArsService::Logout $Cx2;                                  # client-child does a 'logout' from server2
  ArsService::Login $Cx2 -wait LOGI User3 Pwd3 Server3 Enc3;# client-child login into server3
  ArsService::DoSomeWork $Cx2 -wait DOSW ....               # client-child is doing some work on server3
  ArsService::DoSomeWork2 $Ctx -wait DSW2 ....              # client-parent is doing some additional work on server1
  ArsService::Logout $Cx2;                                  # client-child does a 'logout' from server3
  rename $Cx2 "";                                           # client-child destroy
  ArsService::Logout $Ctx;                                  # client-parent does a 'logout' from server1
  rename $Ctx "";                                           # client-parent destroy and stop >>ArsService<<
  

11. Do services have to be locked so that they can complete processing complete requests

• DEFINITION: In this case locked means prevented from execution.
• libFreiburg 2.0 was designed to act as an event based scheduler doing io in a parallel manner
  
• Example: multiple parent-context (CTX1, CTX2, CTX3) : using 'Tcl' code

  # SLEP ... service doing a 'sleep' system-call on the server -> blocking the server
  # client is doing the following 3 jobs
  after 100 "send $CTX1 -wait SLEP 10";	# test-case: service sleeps 10 seconds
  after 100 "send $CTX2 -wait SLEP 5";	# test-case: service sleeps 5 seconds
  after 100 "send $CTX3 -wait SLEP 8";	# test-case: service sleeps 8 seconds
  # wait on finish all 3 jobs
  

  the example needs 10 seconds to finish and not 23
  
• Example: single parent-context (CTX) : using 'Tcl' code

  # SLEP ... service doing a 'sleep' system-call on the server -> blocking the server
  # client is doing the following 3 jobs
  after 100 "send $CTX -wait SLEP 10";	# test-case: service sleeps 10 seconds
  after 100 "send $CTX -wait SLEP 5";	# test-case: service sleeps 5 seconds
  after 100 "send $CTX -wait SLEP 8";	# test-case: service sleeps 8 seconds
  # wait on finish all 3 jobs
  

  the example needs 23 seconds to finish and not 10

11.a How does a 'service' detect different 'context' requests''

• INTRO: client/server versa parent/child
• parent and child share the same communication interface
• to separate the parent request from a child request the context handle of the current request is part of the communication header (e.g. MyContext versa RemoteContext)
• overview libFreiburg protocol

12. How do they signal success or failure

• INTRO: client/server versa parent/child
• every transaction submit an return-state of (OK='O',WARNING='W',ERROR='E') to the caller
  using 'C' code

  // start: call a service and wait for the answer
  MqErrorCheck (MqSendEND_AND_WAIT (send, token, timeout));
  // read all result items 
  MqReadSP const read = msgque->read;
  while (MqGetNumItems (read)) {
      TclErrorCheck (MqRead? (read, ????)); ...
  }
  // check the return code
  if (MqReadGetReturnCode(read) != 'O') {
      // something is wrong ...
      MqReadSP readRef;
      int retNum = MqReadGetReturnNum(read);
      char retCode = MqReadGetReturnCode(read);
      MqReadCreate_RET_Ref (read, &readRef);
      // read all return-message items
      while (MqGetNumItems (read)) {
          TclErrorCheck (MqRead? (read, ????)); ...
      }
      MqReadDelete_RET_Ref (&readRef);
      // return ERROR
  }
  // everything is OK return result items
  

  using 'Tcl' code

  proc ::TclExceptionProc { Ctx Token RetCode RetNum RetList } {
      set msg "\[$Ctx\] Token:$Token, RetNum:$RetNum, RetText:[join $RetList -]"
      if {$RetCode eq "W"} {
  	puts stderr $msg
      } else {
  	error $msg
      }
  }
  set Ctx [msgque create ... -exception ::TclExceptionProc ...]
  $Ctx send -wait TEST ....
  

• every stdout/stderr message will be available for debugging purpose

prev: using threads - top: libFreiburg 2.0 - next: libFreiburg 2.0

---