FileMaker Server 11 includes several components to publish databases to client machines. The Database Server is perhaps the most obvious component, as this hosts the databases that are shared with clients.
Any type of web publishing requires the Web Publishing Engine (WPE) to be active and running. Custom web publishing such as PHP communicates to the Database Server via the WPE. But what happens when a web page is no longer is responsive? Which process failed? Is it a page error, a Database Server problem, or an issue with the WPE? As you can imagine, manually checking the state of the server becomes tedious.
The ability to build an automatic monitoring method becomes critical for anyone managing web pages that communicate with FileMaker Server, to rapidly identify problems and minimize downtime.
The Admin Console is a GUI-based tool to manage databases, backup and script schedules, clients, statistics, and the current state of the database and WPE processes. However, as it is a GUI-based system, it needs eyes-on management. In order to automate the process we need to turn to some system-based tools.
Turning to the Command Line
One oft-used method is to create a PHP page that pings the Database Server. If the ping – a PHP request – returns a valid result, then all is well. If the ping returns an error, the administrator intervenes. However, when setting this up, I ran into issues with PHP sendmail failures, and I wanted a process that minimized manual intervention.
To fully automate the process I used one set of tools to check the state of the WPE, and another to automatically switch on the WPE if it wasn’t running. Although FileMaker Server comes with a command line interface – fmsadmin – there is no documented process to manage the WPE in FileMaker Server 11.
When FileMaker, Inc. released Server 12 on April 4, 2012, new options were added to start and stop processes; now you can stop and start the WPE process via fmsadmin. However, if you’re running FileMaker Server 11, you need to take advantage of some undocumented processes. (Here the usual caveat applies: test to make sure this works under your circumstances).
In the Mac OS X environment you can use Unix commands to check running processes. One such command “nc” lets you scan for listening daemons to check if a process is running on a specific port, without sending data. By scanning a WPE port you then can see if this is active for a given IP address.
nc -zw 3 127.0.0.1 16008
This command will return nothing if the process is inactive. If the process is running, the command returns a text string with a success message.
Once we’re able to run such a command in the Terminal, the next step is to set this up in a shell script. If we first check for the WPE and find that it isn’t running, we next verify that the Database Server is running. Once we know this process is active, we then run a sequence of fmsadmin commands. These I gleaned from a website (thanks to Andrew Duncan from Databuzz) that provides a preference pane to control FileMaker Server 11. The key command is:
fmsadmin start|stop adminserver|wpc|cwp
When I tried this out in the Terminal, I found that it only would work for me when I ran them on separate lines, and in a certain sequence.
fmsadmin start wpc fmsadmin start cwp
After all this is combined into a shell script, you can run this as a cron job on Mac OS X. I simply saved the code below into a wpe.sh file, made this executable and set up the cron to check the process. The #lines are comments that tell me what happens inside each branch, and the if statements check to see whether there is a value returned from the query or not.
#!/bin/bash #wpe running up_wpe=`nc -zw 3 127.0.0.1 16008` if [ -z "$up_wpe" ]; then #echo "wpe is not running" up_fms=`nc -zw 3 127.0.0.1 5003` if [ -n "$up_fms" ]; then #echo "DBS running" fmsadmin start wpc fmsadmin start cwp fi fi
This probably is an un-orthodox method to check the WPE process and start it up, but it appears to work. With FileMaker Server 12 you still need to step outside FileMaker Server’s built-in tools to check the process, but you’re able to run the documented “fmsadmin start wpe” command. Possibly there are other ways to monitor the WPE process, and refinements to this method. We’d love to hear any that have worked for you.
Nice post !
I think there are two semicolumns missing after both if statements.
if [ -z "$up_wpe" ]; then #echo "wpe is not running"
if [ -n "$up_fms" ]; then #echo "DBS running"
Thanks. It seems to work both with and without the semi-colons, so perhaps these are optional rather than required in this case?
Thank you! This problem has been frustrating me for many months.
I run the older Filemaker Pro 10 Server which can’t make use of the fmsadmin command.
As a work-around, I developed this small script based on your post:
#!/bin/bash
nc -zw 3 127.0.0.1 16018 1>/dev/null 2>&1; result=$?;
if [ $result = 1 ]; then
/Library/FileMaker\ Server/Data/Scripts/restart_wpe.app/Contents/MacOS/restart_wpc
fi
The restart_wpe.app is a small Automator app (MacOS X) that records my mouse movement as I stop and restart the web publishing via the Filemaker Server Admin Console.
Notice also that I used port 16018 as this is the AJP port used to connect the Web Server to the WPC, and seemed to be the main one failing consistently.
I’ve scheduled the script to run every 30 minutes, and so far so good.
Matt – You are very welcome. I am glad you found the article helpful.
I get error at line 6 – unexpected end of line.
Any suggestions?
Adam – Unfortunately, we are no longer using FileMaker Server 11 in a production environment so I cannot help.
While my initial solution to this problem worked for a while, it wasn’t long before it recurred. Using netstat to monitor the ports showed they were still open but that Tomcat wouldn’t release sessions after they expired. Increased the acceptCount in the tomcat server.xml, extended the keepAlive times, and slimmed down the workers.properties to just use wpc1 and wpc2 (as I’m not using custom web publishing) to give me some breathing room, but the problem remained.
I tried a different tactic and monitored the log entries, and used the tomcat error entry to trigger my automator restart_wpe.app script. The shell script below is scheduled to run every 15 minutes from the FileMaker Admin Console:
#!/bin/bash
tail -n1 /Library/FileMaker\ Server/Logs/web_server_module_log.txt | grep tomcat 1>/dev/null 2>&1; result=$?;
if [ $result != 1 ]; then
/Library/FileMaker\ Server/Data/Scripts/restart_wpe.app/Contents/MacOS/restart_wpc
fi
exit
So far this seems to be working better, although I still need to manually restart the web publishing service manually on occasion. Hopefully thats helpful to someone.
FileMaker Server 10.0.2.206
Mac OS X 10.5.8
Matt – Thanks for sharing your solution.