Search · Index

AOLserver administration

AOLserver administration

automating AOLserver startup and stop

The simplest way to start and stop and OpenACS site is to run the startup shell script provided in the OpenACS distribution, /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/run. This runs as a regular task, and logs to the logfile. To stop the site, kill the script.

A more stable way to run OpenACS is with a "keepalive" mechanism of some sort, so that whenever the server halts or is stopped for a reset, it restarts automatically. This is recommended for development and production servers.

The Reference Platform uses Daemontools to control AOLserver. A simpler method, using init, is next.

Configuring Daemontools

1. Daemontools must already be installed. If not, install it.

2. Each service controlled by daemontools must have a directory in /service. That directory must have a file called run. It works like this:

   • The init program starts every time the computer is booted.

   • A line in init's configuration file, /etc/inittab, tells init to run, and to restart if necessary, svscanboot.

   • svscanboot checks the directory /service every few seconds.

   • If it sees a subdirectory there, it looks for a file in the subdirectory called run. If it finds a run file, it creates a supervise process

   • supervise executes the run script. Whenever the run script stops, supervise executes it again. It also creates additional control files in the same directory.

   Hence, the AOLserver instance for your development server is started by the file /service/$OPENACS_SERVICE_NAME/run. But we use a symlink to make it easier to add and remove stuff from the /service, so the actual location is /var/lib/aolserver/$OPENACS_SERVICE_NAMEetc/daemontools/run.

   Daemontools creates additional files and directories to track status and log. A daemontools directory is included in the OpenACS tarball at /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools. To use it, first kill any existing AOLserver instances. As root, link the daemontools directory into the /service directory. Daemontools' svscan process checks this directory every five seconds, and will quickly execute run.

   [$OPENACS_SERVICE_NAME etc]$ killall nsd
   nsd: no process killed
   [$OPENACS_SERVICE_NAME etc]$ emacs /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/run
   [$OPENACS_SERVICE_NAME etc]$ exit
   
   [root root]# ln -s /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/daemontools/run /service/$OPENACS_SERVICE_NAME/run

   Verify that AOLserver is running.

   [root root]# ps -auxw | grep nsd
   $OPENACS_SERVICE_NAME   5562 14.2  6.2 22436 15952 ?       S    11:55   0:04 /usr/local/aolserver/bin/nsd -it /var/lib/aolserver/$OPENACS_SERVICE_NAME/etc/config.tcl -u serve
   root      5582  0.0  0.2  3276  628 pts/0    S    11:55   0:00 grep nsd
   [root root]#

3. The user $OPENACS_SERVICE_NAME can now control the service $OPENACS_SERVICE_NAME with these commands:

   • svc -d /service/$OPENACS_SERVICE_NAME - Bring the server down

   • svc -u /service/$OPENACS_SERVICE_NAME - Start the server up and leave it in keepalive mode.

   • svc -o /service/$OPENACS_SERVICE_NAME - Start the server up once. Do not restart it if it stops.

   • svc -t /service/$OPENACS_SERVICE_NAME - Stop and immediately restart the server.

   • svc -k /service/$OPENACS_SERVICE_NAME - Sends the server a KILL signal. This is like KILL -9. AOLserver exits immediately. If svc -t fails to fully kill AOLserver, use this option. This does not take the server out of keepalive mode, so it should still bounce back up immediately.

4. Install a script to automate the stopping and starting of AOLserver services via daemontools. You can then restart a service via restart-aolserver $OPENACS_SERVICE_NAME

   [root root]# cp /var/lib/aolserver/$OPENACS_SERVICE_NAME/packages/acs-core-docs/www/files/restart-aolserver-daemontools.txt /usr/local/bin/restart-aolserver
   [root root]# chmod 755 /usr/local/bin/restart-aolserver
   [root root]#

5. At this point, these commands will work only for the root user. Grant permission for the web group to use svc commands on the $OPENACS_SERVICE_NAME server.

   [root root]# /usr/local/bin/svgroup web /service/$OPENACS_SERVICE_NAME
   [root root]#

6. Verify that the controls work. You may want to tail -f /var/lib/aolserver/$OPENACS_SERVICE_NAME/log/$OPENACS_SERVICE_NAME-error.log in another window, so you can see what happens when you type these commands.

   Most of this information comes from Tom Jackson's AOLserver+Daemontools Mini-HOWTO.

Table 6.1. How it Works

Program	Invoked by this program ...	... using this file	Where to find errors	Log goes to	Use these commands to control it
svscanboot	init	/etc/inittab	ps -auxw | grep readproctitle	n/a
aolserver	supervise (a child of svscanboot)	/service/$OPENACS_SERVICE_NAME/run	/var/lib/aolserver/$OPENACS_SERVICE_NAME/log/error.log	/var/lib/aolserver/$OPENACS_SERVICE_NAME/log/$OPENACS_SERVICE_NAME.log	svc -k /service/$OPENACS_SERVICE_NAME
postgresql	Redhat init scripts during boot	/etc/init.d/postgresql	/usr/local/pgsql/data/server.log		service postgresql start (Red Hat), /etc/init.d/postgresql start (Debian)

init - keeping AOLserver alive

This is an alternative method for keeping the AOLserver process running. The recommended method is to run AOLserver supervised.

This step should be completed as root. This can break every service on your machine, so proceed with caution.

• There are 2 general steps to getting this working.

  1. Install a script called restart-aolserver. This script doesn't actually restart AOLserver - it just kills it.

  2. Ask the OS to restart our service whenever it's not running. We do this by adding a line to /etc/inittab.

  Calling restart-aolserver kills our service. The OS notices that our service is not running, so it automatically restarts it. Thus, calling restart-aolserver effectively restarts our service.

• This script needs to be SUID-root, which means that the script will run as root. This is necessary to ensure that the AOLserver processes are killed regardless of who owns them. However the script should be executable by the web group to ensure that the users updating the web page can use the script, but that general system users cannot run the script. You also need to have Perl installed and also a symbolic link to it in /usr/local/bin.

  [joeuser ~]$ su - 
  Password: ***********
  [root ~]# cp /var/lib/aolserver/service0/packages/acs-core-docs/www/files/restart-aolserver.txt /usr/local/bin/restart-aolserver
  [root ~]# chown root.web /usr/local/bin/restart-aolserver
  [root ~]# chmod 4750 /usr/local/bin/restart-aolserver
  [root ~]# ln -s /usr/bin/perl /usr/local/bin/perl
  [root ~]# exit

• Test the restart-aolserver script. We'll first kill all running servers to clean the slate. Then, we'll start one server and use restart-aolserver to kill it. If it works, then there should be no more servers running. You should see the following lines.

  [joeuser ~]$ killall nsd
  nsd: no process killed
  [joeuser ~]$ /usr/local/aolserver/bin/nsd-postgres -t ~/var/lib/aolserver/service0/nsd.tcl
  [joeuser ~]$ restart-aolserver service0
  Killing 23727 
  [joeuser ~]$ killall nsd
  nsd: no process killed

  The number 23727 indicates the process id(s) (PIDs) of the processes being killed. It is important that no processes are killed by the second call to killall. If there are processes being killed, it means that the script is not working.

• Assuming that the restart-aolserver script worked, login as root and open /etc/inittab for editing.

  [joeuser ~]$ su -
  Password: ************
  [root ~]# emacs -nw /etc/inittab

• Copy this line into the bottom of the file as a template, making sure that the first field nss1 is unique.

  nss1:345:respawn:/usr/local/aolserver/bin/nsd-postgres -i -u nobody -g web -t /home/joeuser/var/lib/aolserver/service0/nsd.tcl

• Important: Make sure there is a newline at the end of the file. If there is not a newline at the end of the file, the system may suffer catastrophic failures.

• Still as root, enter the following command to re-initialize /etc/inittab.

  [root ~]# killall nsd    
  nsd: no process killed
  [root ~]# /sbin/init q

• See if it worked by running the restart-aolserver script again.

  [root ~]# restart-aolserver service0
  Killing 23750

If processes were killed, congratulations, your server is now automated for startup and shutdown.

external monitoring of server uptime

http://openacs.org/doc/uptime.html

starting aolserver before postgresql

Add a pause to the aolserver "run" script, for example:

    sleep 15

Alternately, John Sequeira suggests adding this code to the run script:

pid=`pidof -s postmaster`
while !([ $pid ] && /etc/init.d/postgresql status > /dev/null 2>&1)
do
pid=`pidof -s postmaster`
echo "checking...not yet"
sleep 1
done

run AOLserver on port 80 (or <1024) To run AOLserver on a port below 1024 (normally, for a webserver use port 80), edit the run script. /var/lib/aolserver/service0/etc/daemontools/run script according to the documentation found there (namely: Add the -b yourip:yourport switch)

Troubleshooting AOLserver

Check the serverlog 

To set up real-time monitoring of the AOLserver error log, from the shell, type

less /var/lib/aolserver/$OPENACS_SERVICE_NAME/log/openacs-dev-error.log

F to show new log entries in real time (like tail -f)
C-c to stop and F to start it up again. 
G goes to the end.
? searches backward 
/ searches forward.  

Cannot view sample AOLserver welcome page There may be a problem with the server configuration. Start by viewing the AOLserver log, which is in /usr/local/aolserver/log/server.log. You should also try to find lines of the form:

[01/Jun/2000:12:11:20][5914.2051][-nssock-] Notice: nssock: listening on http://localhost.localdomain:8000 (127.0.0.1:8000)
[01/Jun/2000:12:11:20][5914.2051][-nssock-] Notice: accepting connections

If you can find these lines, try entering the URL the server is listening on into your browser. If you cannot find these lines, there must be an error somewhere in the file. Search for lines beginning with the word Error instead of Notice.

The sample-config.tcl file grabs your address and hostname from your OS settings.

set hostname        [ns_info hostname]
set address         [ns_info address]

If you get an error that nssock can't get the requested address, you can set these manually. If you type 0.0.0.0, AOLserver will try to listen on all available addresses. Note: ns_info address doesn't appear to be supported in current versions of AOLserver.

set hostname        [ns_info hostname]
#set address         [ns_info address]
set address 0.0.0.0

NsTclInitObjs: sizeof(int) < sizeof(long) This is a 64-bit compile problem that keeps Aolserver from starting. An error detection check in file nsd/tclobj.c is causing the error, and needs to be commented out. Wrap the following code with the C style /* */ comments (as shown), then rebuild. See http://openacs.org/forums/message-view?message_id=309887 for more info.

/*
    if (sizeof(int) < sizeof(long)) {
        Tcl_Panic("NsTclInitObjs: sizeof(int) < sizeof(long)");
    }
*/

Cannot view the OpenACS login or OpenACS Installation: Welcome page. View your error log (/var/lib/aolserver/$OPENACS_SERVICE_NAME/log/$OPENACS_SERVICE_NAME-error.log) to make sure the service is starting without any problems. The most common errors here are trying to start a port 80 server while not root, failing to connect because of a firewall, and aolserver failing to start due to permissions errors or missing files. If you need to make changes, don't forget to kill any running servers with killall nsd.

Modifying AOLserver configuration config.tcl

• httpport - If you want your server on a different port, enter it here. The Reference Platform port is 8000, which is suitable for development use. Port 80 is the standard http port - it's the port used by your browser when you enter http://yourserver.test. So you should use port 80 for your production site.

• httpsport - This is the port for https requests. The Reference Platform https port is 8443. If http port is set to 80, httpsport should be 143 to match the standard.

• address - The IP address of the server. If you are hosting multiple IPs on one computer, this is the address specific to the web site. Each virtual server will ignore any requests directed at other addresses.

• server - This is the keyword that, by convention, identifies the service. It is also used as part of the path for the service root, as the name of the user for running the service, as the name of the database, and in various dependent places. The Reference Platform uses $OPENACS_SERVICE_NAME.

• db_name - In almost all cases, this can be kept as a reference to $server. If for some reason, the tablespace you are using is different than your servername, then you can set it here. You should have a good reason for doing this.

• servername - This is just a *pretty* name for your server.

• user_account - The account that will both own OpenACS files and connect to the database (for Postgresql).

• debug - Set to true for a very verbose error log, including many lines for every page view, success or failure.

AOLserver is very configurable. These are some of the more commonly changed settings. For more options, read the AOLserver docs.

AOLserver command line

See http://www.aolserver.com/docs/admin/tech.html