Asterisk或者FreePBX出現故障如何排查是工程師最基本的技能之一。如何排查復雜的底層問題是技術人員經常遇到的問題，如果技術人員能夠提供非常完整的debug消息，這樣就會大大提高排查問題的速度，提高工作效率可以快速和其他支持人員溝通。
在Asterisk和FreePBX環境下，官方提供了多種排查的方式，通過價值排查方式可以輕松解決這些問題，用戶可以排查應用級FreePBX和界面的問題，也可以排查Asterisk平臺級的問題（系統崩潰，死鎖等）。 主要的排查方式包括：
- FreePBX CLI Debug
- FreePBX Debug
- Browser Debug （Console Log）
- CHROME
- SAFARI
- IE9:
- FIREFOX
- OPERA
- Asterisk Logs
Backtraces （Segfaults/Core Dumps/Asterisk Crashing）
1）通過執行命令來激活Debug方式，保存相關的日志到相應的默認路徑。 在linux CLI環境下，執行amportal a dbug，顯示以下輸出結果， 開發人員可以根據相應的路徑文件檢查問題。
[root@dahdidev1 ~]# amportal a dbug
Fetching FreePBX settings with gen_amp_conf.php
trying to run as user asterisk:
[FPBXDBUGDISABLE] already set to [0]
==> /var/log/asterisk/freepbx_dbug <==
==> /var/log/httpd/error_log <==
mv: cannot stat `/var/lib/asterisk/agi-bin/license-*.zl': No such file or directory
[Wed Dec 03 00:05:09 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/ucp/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapi
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /tftpboot/zc
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/aastra
[Wed Dec 03 01:14:26 2014] [error] [client 125.64.35.67] File does not exist: /var/www/html/restapps
[Wed Dec 03 01:59:48 2014] [error] [client 94.102.49.82] script '/var/www/html/recordings/index.php' not found or unable to stat
[Wed Dec 03 01:59:50 2014] [error] [client 94.102.49.82] File does not exist: /var/www/html/admin/modules/admindashboard
[Wed Dec 03 01:59:51 2014] [error] [client 94.102.49.82] client denied by server configuration: /var/www/html/admin/page.framework.php
==> /var/log/asterisk/freepbx.log <==
[2014-Dec-03 09:44:02] [PHP-WARNING] （/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0） - No license for this product （PBXact） - make sure zend_loader.license_path is properly configured in your ini file!
[2014-Dec-03 09:44:02] [PHP-WARNING] （/var/www/html/admin/modules/endpoint/functions.inc/functions_epm_ucp.php:0） - License check failed!
[2014-Dec-03 09:57:01] [WARNING] （libraries/modulefunctions.legacy.php:7） - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:01] [PHP-NOTICE] （/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53） - Undefined index: register
[2014-Dec-03 09:57:02] [WARNING] （libraries/modulefunctions.legacy.php:7） - Depreciated Function _module_distro_id detected in /var/www/html/admin/modules/sysadmin/functions.inc/general.php on line 107
[2014-Dec-03 09:57:02] [PHP-NOTICE] （/var/www/html/admin/modules/sysadmin/functions.inc/license.php:53） - Undefined index: register
[2014-Dec-03 09:58:20] [INFO] （bin/freepbx_setting:39） - trying to run as user asterisk:
[2014-Dec-03 09:58:20] [INFO] （bin/freepbx_setting:40） -
[2014-Dec-03 09:58:21] [INFO] （bin/freepbx_setting:61） - [FPBXDBUGDISABLE] already set to [0]
[2014-Dec-03 09:58:21] [INFO] （bin/freepbx_setting:62） -
2）Browser Debug （Console Log），這個方法是通過瀏覽器的開放模式來檢查問題。開發人員需要根據自己所使用的瀏覽器，通過不同的命令方式來排查。
CHROME:
- Press either CTRL + SHIFT + J to open the “console” tab of the Developer Tools.
- Alternative method:
- Press either CTRL + SHIFT + I or F12 to open the Developer Tools.
- Press ESC （or click on “Show console” in the bottom right corner） to slide the console up.
- Note: In Chrome’s dev tools, there is a “console” tab. However, a smaller “slide-up” console can be opened while any of the other tabs is active.
SAFARI:
- Press CTRL + ALT + I to open the Web Inspector.
- See Chrome’s step 2. （Chrome and Safari have pretty much identical dev tools.）
- Note: Step 1 only works if the “Show Develop menu in menu bar” check box in the Advanced tab of the Preferences menu is checked!
IE9:
- Press F12 to open the developer tools.
- Click the “console” tab.
FIREFOX:
Press CTRL + SHIFT + K to open the Web console （COMMAND + SHIFT + K on Macs）。
or, if Firebug is installed:
Press F12 to open Firebug.
Click on the “console” tab.
OPERA:
Press CTRL + SHIFT + I to open Dragonfly.
Click on the “console” tab.
3） Asterisk Logs 日志文件
root@freepbxdev1 ivr]# tail -f /var/log/asterisk/full // 默認asterisk日志存儲路徑
[2014-12-03 13:20:53] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:22:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:23:22] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:25:51] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:27:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:27:59] WARNING[6287] res_calendar_caldav.c: Unknown response to CalDAV calendar myGoogleCal, request REPORT to /calendar/dav/en.usa%23holiday%40group.v.calendar.google.com/events: SSL handshake failed: SSL error: GnuTLS internal error.
[2014-12-03 13:28:20] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:30:49] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:32:41] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
[2014-12-03 13:33:18] WARNING[6227] res_pjsip_mwi.c: AOR 1000 has no configured mailboxes. MWI subscription failed
如果需要查找特定的條件的話，也可以通過以下命令grep排查報錯日志信息：
[root@34693894 ~]# grep 1518526777.67 /var/log/asterisk/full*
- var/log/asterisk/full-20180214:[2018-02-13 08:59:37] VERBOSE[24184][C-00000001] pbx.c: Executing [s@macro-user-callerid:1] Set（"SIP/5002-00000001", "TOUCH_MONITOR=1518526777.67"） in new stack
- var/log/asterisk/full-20180214:[2018-02-13 08:59:37] VERBOSE[24184][C-00000001] pbx.c: Executing [700@from-internal:37] QueueLog（"SIP/5002-00000001", "700,1518526777.67,NONE,DID,"） in new stack
4 可以使用命令查找方式：
ls -l /tmp | grep core. // 找到相關的core dump 信息。
5 可以使用Sangoma 的開發排查根據來排查，用戶需要安裝這些開發工具，安裝的是ISO版本：
# 僅支持 FreePBX Distro 7!
yum install -y sangoma-devel
debuginfo-install --enablerepo=centos7-debuginfo asterisk14 // 這里的支持的是asterisk 14，用戶可以使用不同的版本，例如15來排查Asterisk15.
6 如果用戶運行 13.14.0 或者 14.3.0以上版本，也可以使用系統默認支持的工具：
/var/lib/asterisk/scripts/ast_coredumper /tmp/[name of the core file]
7 使用最簡單命令獲得崩潰信息：
gdb -se "asterisk" -ex "bt full" -ex "thread apply all bt" --batch -c [name of the core file] > /tmp/backtrace.txt
8 Asterisk新排查利器：scripts/ast_coredumper 命令
此命令是13.14.0 和 14.3.0以上版本添加到Asterisk崩潰系統命令，功能非常強大，可以輕松排查各種asteisk問題，死鎖，收集系統日志信息。配置文件在默認Asterisk路徑：etc/asterisk/ast_debug_tools.conf。具體命令使用方式如下：
var/lib/asterisk/scripts/ast_coredumper --help
AME
ast_coredumper - Dump and/or format asterisk coredump files
SYNOPSIS
ast_coredumper [ --help ] [ --running | --RUNNING ] [ --latest ]
[ --tarball-coredumps ] [ --delete-coredumps-after ]
[ --tarball-results ] [ --delete-results-after ]
[ --tarball-uniqueid="<uniqueid>" ]
[ --no-default-search ] [ --append-coredumps ]
[ <coredump> | <pattern> … ]
DESCRIPTION
Extracts backtraces and lock tables from Asterisk coredump files.
For each coredump found, 4 new result files are created:
- <coredump>.brief.txt: The output of "thread apply all bt".
- <coredump>.thread1.txt: The output of "thread apply 1 bt full".
- <coredump>.full.txt: The output of "thread apply all bt full".
- <coredump>.locks.txt: If asterisk was compiled with
"DEBUG_THREADS", this file will contain a dump of the locks
table similar to doing a "core show locks" from the asterisk
CLI.
Optional features:
- The running asterisk process can be suspended and dumped.
- The coredumps can be merged into a tarball.
- The coredumps can be deleted after processing.
- The results files can be merged into a tarball.
- The results files can be deleted after processing.
Options:
--help
Print this help.
--running
Create a coredump from the running asterisk instance and
process it along with any other coredumps found （if any）。
WARNING: This WILL interrupt call processing. You will be
asked to confirm.
--RUNNING
Same as --running but without the confirmation prompt.
DANGEROUS!!
--latest
Process only the latest coredump from those specified （based
on last-modified time）。 If a dump of the running process was
requested, it is always included in addition to the latest
from the existing coredumps.
--tarball-coredumps
Creates a gzipped tarball of all coredumps processed.
The tarball name will be:
/tmp/asterisk.<timestamp>.coredumps.tar.gz
--delete-coredumps-after
Deletes all processed coredumps regardless of whether
a tarball was created.
--tarball-results
Creates a gzipped tarball of all result files produced.
The tarball name will be:
/tmp/asterisk.<timestamp>.results.tar.gz
--delete-results-after
Deletes all processed results regardless of whether
a tarball was created. It probably doesn't make sense
to use this option unless you have also specified
--tarball-results.
--tarball-uniqueid="<uniqueid>"
Normally DATEFORMAT is used to make the tarballs unique
but you can use your own unique id in the tarball names
such as the Jira issue id.
--no-default-search
Ignore COREDUMPS from the config files and process only
coredumps listed on the command line （if any） and/or
the running asterisk instance （if requested）。
--append-coredumps
Append any coredumps specified on the command line to the
config file specified ones instead of overriding them.
<coredump> | <pattern>
A list of coredumps or coredump search patterns. Unless
--append-coredumps was specified, these entries will override
those specified in the config files.
Any resulting file that isn't actually a coredump is silently
ignored. If your patterns contains spaces be sure to only
quote the portion of the pattern that DOESN'T contain wildcard
expressions. If you quote the whole pattern, it won't be
expanded.
If --no-default-search is specified and no files are specified
on the command line, then the only the running asterisk process
will be dumped （if requested）。 Otherwise if no files are
specified on the command line the value of COREDUMPS from
ast_debug_tools.conf will be used. Failing that, the following
patterns will be used:
/tmp/core[-._]asterisk!（*.txt）
/tmp/core[-._]$（hostname）�。�*.txt）
NOTES
You must be root to use ast_coredumper.
The script relies on not only bash, but also recent GNU date and
gdb with python support. *BSD operating systems may require
installation of the 'coreutils' and 'devel/gdb' packagess and minor
tweaking of the ast_debug_tools.conf file.
Any files output will have ':' characters changed to '-'. This is
to facilitate uploading those files to Jira which doesn't like the
colons.
FILES
/etc/asterisk/ast_debug_tools.conf
~/ast_debug_tools.conf
./ast_debug_tools.conf
#
# This file is used by the Asterisk debug tools.
# Unlike other Asterisk config files, this one is
# "sourced" by bash and must adhere to bash semantics.
#
# A list of coredumps and/or coredump search patterns.
# Bash extended globs are enabled and any resulting files
# that aren't actually coredumps are silently ignored
# so you can be liberal with the globs.
#
# If your patterns contains spaces be sure to only quote
# the portion of the pattern that DOESN'T contain wildcard
# expressions. If you quote the whole pattern, it won't
# be expanded and the glob characters will be treated as
# literals.
#
# The exclusion of files ending ".txt" is just for
# demonstration purposes as non-coredumps will be ignored
# anyway.
COREDUMPS=（/tmp/core[-._]asterisk!（*.txt） /tmp/core[-._]$（hostname）�。�*.txt））
# Date command for the "running" coredump and tarballs.
# DATEFORMAT will be executed to get the timestamp.
# Don't put quotes around the format string or they'll be
# treated as literal characters. Also be aware of colons
# in the output as you can't upload files with colons in
# the name to Jira.
#
# Unix timestamp
#DATEFORMAT='date +%s.%N'
#
# *BSD/MacOS doesn't support %N but after installing GNU
# coreutils…
#DATEFORMAT='gdate +%s.%N'
#
# Readable GMT
#DATEFORMAT='date -u +%FT%H-%M-%S%z'
#
# Readable Local time
DATEFORMAT='date +%FT%H-%M-%S%z'
以上命令需要asterisk的相關源代碼編譯參數，例如 DEBUG_THREADS等相關參數。
通過以上幾種排查方式，技術人員可以輕松排查各種IPPBX界面以及應用方面的問題，也可以輕松排查Asterisk底層系統的崩潰，死鎖問題。特別是Asterisk系統底層的工具給開發人員帶來了極大的便利。
參考資料：
https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace
http://freepbx.org.cn/wiki/index.php?title=FreePBX
關注微信公眾號：asterisk-cn，獲得有價值的Asterisk行業分享
Asterisk freepbx 中文官方論壇：http://bbs.freepbx.cn/forum.php
Asterisk freepbx技術文檔： www.freepbx.org.cn
融合通信商業解決方案，協同解決方案首選產品：www.hiastar.com
Asterisk/FreePBX中國合作伙伴，官方qq技術分享群（3000千人）：589995817
