[asterisk-users] Re : asterisk-users] Strange and complete failure of Asterisk 1.8

[Antoine Megalla]

Well,

I had exactly the same issue as you described.

It turned out to be a piece of malicious software that was running on the server.

The customer server was compromised due to a weak root password and only Asterisk process was the target of the malicious program that was embedded deep into the server.

The exact details escape me, but I do remember that it took more than two days of tracing and conducting security forensics to locate the exact cause of asterisk totally failing (I remember doing some GDB and kernel level syscall tracing with the kernel symbols installed, it was an educational adventure...)

The problem was that even when I did a recompile and fresh installation, the malicious software would still target the new asterisk executable.

The attacking software was complicated, hard to detect and almost impossible to remove.

When I realized that the server was deeply compromised, I reinstalled CENTOS from scratch on the same hardware, hardened the root password and that was the end of this issue.

I hope this might save you some frustration.

Take care,
Antoine Megalla



Date: Wed, 27 May 2015 13:55:22 +0200
From: "Stefan Viljoen" <viljoens at verishare.co.za>
To: <asterisk-users at lists.digium.com>
Subject: [asterisk-users] Strange and complete failure of Asterisk 1.8
Message-ID: <006101d09874$030f7f80$092e7e80$@verishare.co.za>
Content-Type: text/plain;    charset="us-ascii"

Hi all

We've had a very strange failure on an Asterisk 1.8 install that has been
running for about a year at a customer site.

The physical hardware is fine, all other services off the Centos 6.5 server
are running. Only Asterisk is not working...

The first symptom was that no calls can be made over the SIP phones used
with it, and no calls could be received over the SIP trunk connected to it.

I checked and noted that 

sip show peers

in the CLI would either do nothing (e. g. just show asterisk*cli> again,
with no response) or it would return only this:

asterisk*CLI> sip show peers
Name/username              Host                                    Dyn
Forcerport ACL Port     Status
asterisk*CLI>

A module show like sip also does literally nothing, just

asterisk*cli> module show like sip
asterisk*CLI>

Soon after this, I lost the capacity to get any response if I do an asterisk
-r on the commandline - it would just hang indefintely.

Did a reboot, and then, I couldnt start asterisk at all - entering

# asterisk

would also just hang.

So, I recompiled asterisk from source and reinstalled the executable and all
the module files. Still the same.

I happened to have an older asterisk executable from a few months before
laying around and sha256summed it - and there was a difference in the
checksum vs. the non-working asterisk binary - BUT it turned out that the
newly recompiled asterisk binary has the SAME SHA256 checksum as the
non-working asterisk binary.

System seems fine otherwise, nothing relevant in /var/log/messages or dmesg
indicating a hardware failure. /var/log/asterisk/messages also contains no
strange warnings or errors.

Anybody got any idea why I cannot resuscitate my Asterisk install, even
after recompiling it from scratch from source? Why would asterisk die like
this in the first place?

Thanks