Discussion:
[Ltsp-discuss] Recipe: Forwarding (remote logging) syslog messages with rsyslogd
John Hupp
2012-12-03 22:54:45 UTC
Permalink
While trying to troubleshoot the problem in which many or most clients
stall during boot at a blank, black screen (which I'm still
troubleshooting), I thought it would be useful to see the syslog
messages from the client. But I was having trouble finding
documentation of how to do that with rsyslog(only syslog-ng). I imagine
that variouspeople here could whip up this configuration in about a
minute (I see now that I failed to ask here), but I had to dig and study
it off-and-on for a couple days.

Here is a working recipe:

To accomplish forwarding of syslog messages by TCP (rather than by UDP
or RELP):

In /etc/rsyslog.conf for the client machine add this to the end of the file:
*.* @@<server's IP address>:10514

For an LTSP network, if it is a standard LTSP 5 setup with a chroot
environment, then just edit the file as above and update the image.

For an LTSP network with an LTSP-PNP setup it is trickier. For
temporary troubleshooting you can just modify /etc/rsyslog.conf, update
the client NBD image, then edit /etc/rsyslog.conf again with the
settings required for the server. If you want a permanent forwarding
setup that will not be overwritten by image updates, then you would
probably write a script in /usr/share/ltsp/init-ltsp.d that modifies
/etc/rsyslog.conf in place on-the-fly during bootup (using the stream
editor command "sed"). But I have not had a successful experience with
that yet, so I merely toss that out as a lead.

-------------------

In /etc/rsyslog.conf for the server where you want the messages
forwarded, un-comment these two lines in rsyslog.conf:
$ModLoad imtcp
$InputTCPServerRun 10514

This much alone on the server will cause forwarded messages to be
received on the server and incorporated in the standard log file at
/var/log/syslog. But they will be added to the messages that are logged
for the server itself. That may do for your purposes since all messages
are tagged with the host name, but you can also have the messages
written to another file by adding these lines to rsyslog.conf:

if $fromhost-ip startswith '192.168.1.' then /var/log/ltspclientlog
& ~

Substitute whatever client IP address applies in your situation. The
above command will separately log received messages from any client with
IP 192.168.1.xxx. Also substitute whatever log name you would like for
my choice of "ltspclientlog."

The configuration above will cause the client syslog messages to be
logged in both /var/log/ltspclientlog and /var/log/syslog. I don't know
why. The "& ~" command is supposed to stop further processing of the
message after it is written to ltspclientlog. Perhaps I misunderstood
the documentation and forwarded messages are *always* written to syslog,
and then perhaps other files as well. Or perhaps LTSP has some
functionality in it that causes forwarded messages to be written to syslog.
Vagrant Cascadian
2012-12-04 02:39:25 UTC
Permalink
was having trouble finding documentation of how to do that with rsyslog (only
syslog-ng). I imagine that various people here could whip up this
configuration in about a minute (I see now that I failed to ask here), but I
had to dig and study it off-and-on for a couple days.
Why not set SYSLOG_HOST=server in lts.conf?

On the server, uncomment the lines in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerRun 514


That uses UDP instead of TCP, but should work fine.


Thanks for the information about splitting the log files into separate files
per client.


live well,
vagrant
John Hupp
2012-12-04 15:43:59 UTC
Permalink
Post by Vagrant Cascadian
was having trouble finding documentation of how to do that with rsyslog (only
syslog-ng). I imagine that various people here could whip up this
configuration in about a minute (I see now that I failed to ask here), but I
had to dig and study it off-and-on for a couple days.
Why not set SYSLOG_HOST=server in lts.conf?
$ModLoad imudp
$UDPServerRun 514
That uses UDP instead of TCP, but should work fine.
Thanks for the information about splitting the log files into separate files
per client.
live well,
vagrant
Setting SYSLOG_HOST is the general-case provision, but the lts.conf
documentation indicates that if SYSLOG_HOST=server is unset, it will
default to directing messages to the LTSP server, which was fine in my
little setup. So I did not set SYSLOG_HOST or SERVER.

By the way, this very setting figured into this paragraph that I wrote
earlier:

The configuration above [/i.e. my recipe/] will cause the client
syslog messages to be logged in both /var/log/ltspclientlog and
/var/log/syslog. I don't know why. The "& ~" command is supposed
to stop further processing of the message after it is written to
ltspclientlog. Perhaps I misunderstood the documentation and
forwarded messages are *always* written to syslog, and then perhaps
other files as well. Or perhaps LTSP has some functionality in it
that causes forwarded messages to be written to syslog.

Which is to say, AFAIK, setting up rsyslog on the server and client as I
indicated should be allthat is needed to set up syslog forwarding. Yet
here is the lts.conf setting for SYSLOG_HOSTwhich also purports to
direct messages, and I don't know what the mechanism is.

--------------------

And if one activates reception of messages by UDP on the server, then in
the client's /etc/rsyslog.conf the single line to add would be:
*.* @<server's IP address>

(This instead of prepending the server address with @@ as I did in my
example for transmission by TCP.)
Matt Johnson
2012-12-07 11:16:41 UTC
Permalink
John,

I'm sure you've tried this, but I should just mention it as I had to do it
to solve a booting problem with some of my clients. Have you added:

ipappend 3

to */var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default*
*
*
*--*
*Matt*
Post by John Hupp
While trying to troubleshoot the problem in which many or most clients
stall during boot at a blank, black screen (which I'm still
troubleshooting), I thought it would be useful to see the syslog messages
from the client. But I was having trouble finding documentation of how
to do that with rsyslog (only syslog-ng). I imagine that various people
here could whip up this configuration in about a minute (I see now that I
failed to ask here), but I had to dig and study it off-and-on for a
couple days.
To accomplish forwarding of syslog messages by TCP (rather than by UDP or
For an LTSP network, if it is a standard LTSP 5 setup with a chroot
environment, then just edit the file as above and update the image.
For an LTSP network with an LTSP-PNP setup it is trickier. For temporary
troubleshooting you can just modify /etc/rsyslog.conf, update the client
NBD image, then edit /etc/rsyslog.conf again with the settings required for
the server. If you want a permanent forwarding setup that will not be
overwritten by image updates, then you would probably write a script in
/usr/share/ltsp/init-ltsp.d that modifies /etc/rsyslog.conf in place
on-the-fly during bootup (using the stream editor command "sed"). But I
have not had a successful experience with that yet, so I merely toss that
out as a lead.
-------------------
In /etc/rsyslog.conf for the server where you want the messages forwarded,
$ModLoad imtcp
$InputTCPServerRun 10514
This much alone on the server will cause forwarded messages to be received
on the server and incorporated in the standard log file at
/var/log/syslog. But they will be added to the messages that are logged
for the server itself. That may do for your purposes since all messages
are tagged with the host name, but you can also have the messages written
if $fromhost-ip startswith '192.168.1.' then /var/log/ltspclientlog
& ~
Substitute whatever client IP address applies in your situation. The
above command will separately log received messages from any client with IP
192.168.1.xxx. Also substitute whatever log name you would like for my
choice of "ltspclientlog."
The configuration above will cause the client syslog messages to be logged
in both /var/log/ltspclientlog and /var/log/syslog. I don't know why. The
"& ~" command is supposed to stop further processing of the message after
it is written to ltspclientlog. Perhaps I misunderstood the documentation
and forwarded messages are *always* written to syslog, and then perhaps
other files as well. Or perhaps LTSP has some functionality in it that
causes forwarded messages to be written to syslog.
------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net
Matthew Wyneken
2012-12-07 12:51:13 UTC
Permalink
While trying to troubleshoot the problem in which many or most clients stall during boot at a blank, black screen (which I'm still troubleshooting),
I just resolved a problem with Ubuntu 12.04 LSTP server and client in which the clients would stall for exactly eight minutes during boot before continuing. If this description matches yours, be sure to look to see if you have a working /etc/resolv.conf. That was my problem. I'm not sure if this is a systematic error in the 12.04 setup or whether I did something wrong to cause resolv.conf to go away.

--
Matthew
John Hupp
2012-12-07 16:03:54 UTC
Permalink
Post by Matthew Wyneken
While trying to troubleshoot the problem in which many or most clients stall during boot at a blank, black screen (which I'm still troubleshooting),
I just resolved a problem with Ubuntu 12.04 LSTP server and client in which the clients would stall for exactly eight minutes during boot before continuing. If this description matches yours, be sure to look to see if you have a working /etc/resolv.conf. That was my problem. I'm not sure if this is a systematic error in the 12.04 setup or whether I did something wrong to cause resolv.conf to go away.
--
Matthew
Matt Johnson also wrote:

I'm sure you've tried this, but I should just mention it as I had to do
it to solve a booting problem with some of my clients. Have you added:

ipappend 3

to /var/lib/tftpboot/ltsp/i386/pxelinux.cfg/default

-------------------------------

Regarding that LTSP client problem that I'm still troubleshooting: One
of the PC's that fails to boot as an LTSP client also has enough
resources to boot the Lubuntu 12.10 Live CD, so I tried that, and I
found that it fails to boot with the same stalled blank, black screen
after the Lubuntu splash screen. I left it like that for ~ 15 minutes
to check for an 8-minute stall, and it didn't budge.

Since a user on the Edubuntu list is troubleshooting the same behavior
with Edu/Ubuntu 12.04 and his LTSP clients, and since I show the same
behavior also on a Lubuntu 12.10 Live CD, I would now suggest that this
is a *buntu 12.04/12.10 problem related to certain chipsets or video chips.
Loading...