I have some Upstart configs used to start some foreground and blocking processes to run in the background as some kind of "daemons", especially those processes do not fork in any case. What I want is to restart them automatically if they exit for any unknown reason, so I configured respawn
, but because those processes don't fork, I DIDN'T configure expect
. That seemed to work as expected, but recently I changed something and today came across something in the Upstart cookbook again which made me wonder...
If you do not specify the expect stanza, Upstart will track the life cycle of the first PID that it executes in the exec or script stanzas.
I do use a script
stanza to build the classpath for my "daemon" and recently added some waiting for Postgres and/or some web applications to become ready and execute my process afterwards using exec
in that script
stanza. For waiting I use tools like ps
and curl
, because I forgot about the "first PID thing" and seem to have confused Upstarts exec
with that of the shell executing the script
.
One example config:
script
waitForPostgres()
{
while [ true ]
do
# http://superuser.com/questions/597549/grep-fails-in-upstart-script
if ps ax | grep "[p]ostgres: wal writer process" > /dev/null
then
break
fi
sleep 10s
done
}
waitForPostgres
cd "$basePath"
CLASSPATH=$basePath/lib
for i in `ls $basePath/lib/*.jar`
do
CLASSPATH=$CLASSPATH:$i
done
export CLASSPATH
exec java [...]
end script
The waitForPostgres
stuff is new and from my understanding everything else are shell built-ins and without waitForPostgres
the first executed and therefore tracked process should be java
. But with my additional function I suspect that Upstart tracks ps
instead and that obviously wouldn't be what I want.
So, which PID is tracked in this example, that of ps
, grep
or java
and why?
And if not java
is tracked, any ideas for workarounds to track that last PID instead of the first?
Thanks!
I found the answer using some slightly different search terms: expect stop
Unfortunately, this means Upstart detects the first invocation of sed as the first PID
Additionally the answer for a workaround is nearby, pre-start
, which is exactly designed for such purpose. Didn't thought about that, even though I've read it before in the docs...
I've found an interesting proposal as well: Possibility to tell explicitly what PID to track
Because of some problems with one of my services I had another look in how it behaves with Upstart and found the following: My "script" stanza is still calling binaries like "dirname", "readlink" and "ls", to not need to hard code some working dir, build some Java classpath and such. The important part is that those are no shell built-ins and therefore should be tracked by Upstart, because "readlink" is the first binary executed in "start". But this is not the case, those PIDs seem to be ignored and Upstart instead tracks the PID of the "exec java ..." command I really want to see tracked. I verified that by calling "service ... status" and comparing the output to "ps axf | grep ..." and both PIDs match. I can properly restart the service and the former found PID is removed and start the service and ps and service ... status report the same PID again.
Two possible explanations: Because "dirname" and Co. are used as command substitution, Upstart recognizes the created subshells by it's own created shell for the "script" stanza and ignores those PIDs by purpose. Else "exec" may return a PID which overrides any former recognized PID. I doubt the latter and think subshells are simply ignored, which is a very nice feature then.
External links referenced by this document: