Using syslog to get caller id name and number from the OBi110

Started by etc6849, October 16, 2011, 01:57:00 PM

Previous topic - Next topic

etc6849

Hi,

I'm developing an open source OBi110 driver for Premise (a free home automation program).

So far, I have caller id number working with Google Voice by parsing the number from the following syslog data:
<7> [SLIC] CID to deliver: '' 17855553301

However, I'm curious if a CID syslog packet looks like this for VOIP providers and PSTN land-line calls (I don't have a land-line or VOIP provider to test, only GV)?

For services that provide a caller's name (e.g. not Google Voice) would the first part of the packet with two apostrophes usually contain the caller's name?

Premise wiki: http://www.cocoontech.com/wiki/Premise

Premise Forum (I'll post my code there once the driver is complete):
http://www.cocoontech.com/forums/index.php?showforum=51

PS:  When you reply be sure to include which firmware of the OBi110 you are using.  

Kiwi Syslog is a free program you can use, or you are welcome to use the perl program below.
use IO::Socket::INET;

# create a socket to serve syslog data to home automation server
$socket = new IO::Socket::INET(LocalPort => '5000', Proto => 'tcp', Listen => 1, Reuse => 1);
die "Could not create socket: $!\n" unless $socket;
print "Waiting for client connection on port 5000.\n";

# wait for client to connect before proceeding
wait_for_client_connection();

#start writing syslog data to client
$syslog = IO::Socket::INET->new(Proto => 'udp', LocalPort => '514');
die "Could not create syslog socket: $!\n" unless $syslog;
print "Syslog server is online.\n";

#if $syslog is defined, then continuously monitor the socket for incoming data
while (defined($syslog))
{
recv($syslog,$msg,1500,0);

# clean up msg some
print $msg;
$msg =~ s/[^\x{20}-\x{7E}]//g;
#$msg =~ s/[^A-Za-z <>0-9\-\.]//g;

# try to send a line to the client (home automation server),
# next we will check result so we know socket is still connected
$resultsend = $client_socket -> send("$msg\r\n");

# if socket is nolonger connected, wait for a new client connection
if (!$resultsend)
{
print "Client connection is lost.\n";
print "Waiting for new client connection...\n";
close $client_socket;
wait_for_client_connection();
}
}
close($syslog);
print $client_socket "Goodbye!\n";
close($socket);

# subroutine that's waits for client connection
sub wait_for_client_connection()
{
# client connection is lost, so wait for reconnect
$clientnotaccepted = 1;
while($clientnotaccepted)
{
# wait for new client connection.
# continue once client is connected...
$client_socket = $socket->accept();
$clientnotaccepted = 0;
}
print "Accepted new client connection.\n";
}


From the FAQs:
Here are the instructions on how to set-up your OBi so that it sends debug messages to the IP address x.y.w.z and port number ppppp. You will replace x.y.w.z and ppppp with the actual IP address and port number where your syslog server program is running.

Step 1: Find your OBi's IP address. From phone handset attached to the OBi, press ***1. The OBi's IP address will be announced.
Step 2: Configure the OBi's parameters for where to send debug messages. Open the OBi's webpage (http://obi-ip-address) and navigate to System Management/Device Admin/Syslog. Fill-in the 'Server' parameter with x.y.w.z, and 'Port' parameter ppppp – if different than default (514). Note that ppppp must be in the range 1 - 65535.
Step 3: Enable logging of SIP Messages for SP1 (or SP2) From the OBi device web page Voice Services settings, go to the SP1 Service (or SP2 Service) settings. Change the 'X_SipDebugOption' parameter to 'Log All Messages.'

Note: If your OBi device is managed from within OBiTALK, you have the option of configuring the syslog parameters from the OBiTALK-based OBi Expert Configuration web pages.


Everton

I cannot answer you specific question, but why not signup for a free Callcentric account or IPcomms or Sipgate (not sure if they have numbers available).  Sipgate and Ipcomms have free incoming.

QBZappy

etc6849,

Hi, welcome

It's exciting to see another OBi app appearing on the scene. Just to keep everything together perhaps you can post in this thread:

3rd party apps development for the OBi 110/100
http://www.obitalk.com/forum/index.php?topic=1648.msg10743#msg10743
Owner of the 1st OBi110/100 units in service in Canada & South America. 1st OBi202 on my street. 1st OBi1032 in Montreal.

infin8loop

I posted a perl syslog script back in August. 

It's here: http://www.obitalk.com/forum/index.php?topic=185.msg9107#msg9107

If I remember correctly it successfully grabbed both GV and AT&T landline calls (at the time).  If the messages haven't been changed in a subsequent release, it probably will still work.  There are a few comments in the code. 
       
At the time it seemed more efficient to leverage off the syslog messages than scrape the xml pages as others had done.  And I'm still not a perl coder, so don't laugh too hard at the code.  I had a text editor open in one window and google to search perl command syntax in other.  Any "bugs" you might find are really features that are intentionally disabled. 

"This has not only been fun, it's been a major expense." - Gallagher

etc6849

infin8loop, your code is very helpful.  So you are able to extract the caller's name, this is good news!  I don't have a landline to test, so could you please clarify one question?

Which packet below is the most reliable for extracting the caller id name?  I have not seen the <7>
  • DAA CND... packet before; does it always occur along with a [SLIC] CID packet?  In other words, is there a definite need for both else if statements in your code?

    <7> [SLIC] CID to deliver: 'XXXXXXXXXXXXXXX' 11231231234
    <7>
  • DAA CND 08310248,1231231234,XXXXXXXXXXXXXXX,,,

    As far as your code, it was very easy to follow; it's good that you commented things.  The perl script I posted here was my first perl script.  I'd recommend you use perl's regular expressions. 

    I haven't used regular expressions in perl much, but my home automation driver uses them to parse the syslog data.  It leaves things much more generic; e.g. you can easily extract what's inside the single quotes, regardless of length.  You can also easily extract numbers, even if their location changes.

etc6849

For a regular expressions example, here's what my parsing script looks like in vbscript (I didn't know perl that well, so I decided to let my home automation program do all of the hard work):
'
' gOBiHAI
' Extracts the parameters out of an OBiHAI syslog packet (only tested on the OBi110).
' Parses sPkt and populates an array called aParms
' Returns the upper bound of aParms
' The i=0,2,4... index of aParms is a string indicating the packet type
' The i=1,3,5... index of aParms is the data pair to data type i-1
'
' Packet types followed by data values:
' State, followed by one of the following integers: 0 = Unknown,1 = OffHook,2=OnHook,3=Ringing
' Status, followed by a string containing miscellaneous information
' ICMState, followed by one of the following integers: 0 = Unknown,1 = OffHook,2=OnHook,3=Ringing
' CallerIDNumber, followed by number, CallerIDName, followed by name
' IMPORTANT NOTE: CallerIDNumber has a lower index than CallerIDName
' Therefore, CallerIDName will be processed last by OBi110.OnChangeOnNewData

function gOBiHAIParser(sPkt, aParms)
dim oRegEx, oMatches, oMatch
dim Tmp
dim bStatusFlag

'grab status packet
bStatusFlag = true

iCmdLength = len(sPkt)

set oRegEx = New RegExp
with oRegEx
.ignoreCase = true
.global = true

' Test for off hook status response
'
' Note offhook will also correspond to in-call
' incall appears in syslog following a call being answered
' if we didn't do this, status would be stuck on "Ringing" during the entire call
' until the phone is hung up.
.pattern = "((offhook)|(off hook)|(state=incall)|(state:incall))"
set oMatches = .Execute(sPkt)
if oMatches.Count = 1  then
' Confirmed it is a valid off hook packet
redim aParms(1)
aParms(0) = "State"
aParms(1) = 1

'do not grab status packet
bStatusFlag = false
end if

' Test for on hook status response
.pattern = "((onhook)|(on hook))"
set oMatches = .Execute(sPkt)
if oMatches.Count = 1  then
' Confirmed it is a valid on hook packet
redim aParms(1)
aParms(0) = "State"
aParms(1) = 2

'do not grab status packet
bStatusFlag = false
end if

' Test for on ring status
.pattern = "((incoming new call)|(start ring))"
set oMatches = .Execute(sPkt)
if oMatches.Count = 1  then
' Confirmed it is a valid on hook packet
redim aParms(1)
aParms(0) = "State"
aParms(1) = 3

'do not grab status packet
bStatusFlag = false
end if

' Test for caller id data
' I assume that the OBi110 will dump name and number in the same packet
' and that a number will always be present, but a name may be blank...
.pattern = "(cid)"
set oMatches = .Execute(sPkt)
if oMatches.Count = 1  then
' Confirmed it is a valid on CID packet, now find number
.pattern = "([0-9]{7,})"
set oMatches = .Execute(sPkt)
if oMatches.Count > 0  then
redim aParms(3)
aParms(0) = "CallerIDNumber"
aParms(2) = "CallerIDName"

if oMatches.Count = 1  then
aParms(1) = oMatches(0)
' Now parse name information
' If name is less than two characters "Unknown Caller" is returned.
.pattern = "('.{2,}')"
set oMatches = .Execute(sPkt)
if oMatches.Count = 0  then
aParms(3) = "Unknown Caller"
else
aParms(3) = oMatches(0)
end if
elseif oMatches.Count = 2 then
' Handles case where a packet containing a phone number in the single quotes is received
' e.g. [SLIC] to deliver: CID '1235551234' 1233334567
aParms(1) = oMatches(1)
aParms(3) = "Unknown Caller" ' Don't return a number as name!
end if

end if
' Do not grab status packet
bStatusFlag = false
end if

' If no other packets are found, populate "Status" information
if bStatusFlag then
' Report Status info from FXS (foreign exchange server) packet
.pattern = "(<7> fxs:)"
set oMatches = .Execute(sPkt)
if oMatches.Count = 1  then
' Confirmed it is an FXS packet
redim aParms(1)
aParms(0) = "Status"
aParms(1) = .Replace(sPkt, "")
end if
bStatusFlag = false
end if
end with

set oRegEx = nothing
set oMatches = nothing
set oMatch = nothing
gOBiHAIParser = ubound(aParms)
end function

infin8loop

etc6849,

Looks like the: "<7>
  • DAA CND 10190152,1231234567,LNAME FNAME    ,,,"
    message only shows up for calls on the LINE port.  It's most likely redundant to the "<7> [SLIC] CID to deliver:" message. When I wrote the script I thought I saw at times when both messages didn't always come in.  But maybe I dreamed that.  I have attached a commented syslog showing test calls I made to the LINE port (AT&T PSTN), SP1 (GV) and SP2 (voip.ms).  The first three calls originated from another AT&T land line not connected to the OBi110. The last call was made from my cell phone to an iNum local access number and then routed to my iNum supplied by voip.ms on SP2.  I answered each call that's listed in the attached syslog after the callerid number and name  (if available) showed up on the cordless phone attached to the OBi110 phone port.  Hopefully the comments in the syslog will tell you everything you need to know.  I had sip debug messages disabled -- just too much info for this exercise.

    And I forgot to mention I masked names, phone numbers, ip addresses, etc. with string values of identical length to the originals.
             
"This has not only been fun, it's been a major expense." - Gallagher

etc6849

Thanks, this is very helpful.  It appears my parsing script will work on land lines and VOIP.ms as is. :)

Do these extra spaces always show up following FNAME?
<7> [SLIC] CID to deliver: 'LNAME FNAME    ' 1231234567 

If so, my script needs to trim extra spaces.

infin8loop

Quote from: etc6849 on October 19, 2011, 06:06:03 AM
Do these extra spaces always show up following FNAME?
<7> [SLIC] CID to deliver: 'LNAME FNAME    ' 1231234567 
If so, my script needs to trim extra spaces.

Regex is definitely the way to go.  I didn't have voip.ms at the time I wrote my script so I didn't see callerid names shorter than 15 characters with trailing spaces as necessary.  Looks like AT&T PSTN calls always show up with the trailing spaces if needed to pad out to 15 characters.  It was my first perl script, so to make things simple for myself I just went with fixed positions.  I didn't think it would hold up in real use. 

If you haven't already you might want to include code to lookup numbers with no callerid name.  Someone posted this link elsewhere in the forums:  http://freecnam.org/dip?q=12146301210  (I put this particular number in the query).  It should return a one line callerid name if the number is found.

   
"This has not only been fun, it's been a major expense." - Gallagher

etc6849

This is a great idea to look up the caller id name...

I plugged several number into the link by replacing the x with the number:
http://freecnam.org/dip?q=1xxxxxxxxxx

None of the numbers I tried showed a company or individual name?  Am I doing something wrong or is the success rate just low for that particular site?

Also thanks for the info on the padding; this is very helpful!

QBZappy

etc6849

Not every voip DID will show up in a serach because DID numbers are usually not automatically listed on a CID CNAM database. In order for a DID number to get listed for it to be pickedup by services which search for this info, such as http://freecnam.org/, you need to make the effort to get it listed. In my case I have an account with voipms. They offered to list my voipms number for me. There are other ways to get your DID number listed that you can do yourself. I just did a lookup of my GV tel number on http://freecnam.org, it was not listed. The only numbers I am almost certain you will find are numbers that are listed with PSTN service, and those that have listed their own voip DID numbers.

You can have a look here as reference info.
http://www.locallytype.com/pages/submit.htm#IYPUS
Owner of the 1st OBi110/100 units in service in Canada & South America. 1st OBi202 on my street. 1st OBi1032 in Montreal.

infin8loop


Good information to know QBZappy.

While probably not anymore reliable for the reasons you mentioned, there is also:

http://cidlookup.com/lookup/2146301210/

The response is a complete webpage.   The actual data is html "encoded" (view source on the page and look at the data between the @@@@ comment blocks to see what I mean.   You can copy and paste that block into:
http://www.opinionatedgeek.com/dotnet/tools/htmlencode/decode.aspx  to see the decoded information.  I know this "decode" can be duplicated in vbscript or whatever. I guess they are just trying to slow down trying to screen scrape the page. But then you'd still have to parse it.  The page mentions an API, but I'm guessing they want money to use that.

My dad's number comes back as simply "TEXAS" as the callerid name when using this site.  It's an AT&T PSTN number he's had for over 30 years.  He's actually older than I am, believe it or not.  Incoming calls on my AT&T PSTN show his actual name (Lastname Firstname) and no, it's not the phone directory showing it, it shows on a standalone callerid box (remember those?) that's not programmable.

Sorry if this is more information that anyone cares to know.  It's already more information than I need to know (grin).

QBZappy, you said "They offered to list my voipms number for me. "... They made me the same offer... for a $10 fee.  Did I just not know the secret Canadian handshake or something? Just because I say, "house" and not "hooose" and I say "bus" and not "booose". When I made my initial deposit to voip.ms it triggered a fraud alert text message from my credit card company.  At first I didn't put 2 and 2 together and thought the text message was a scam because it had no specific details.  So I looked up the fraud number on the bank's website and it wasn't the same as in the text message.  I called that number instead.  Anyway the alert was actually real.  Didn't fit my charging profile I guess.  Then I get my statement and there's a 75 cent "foreign transaction fee".  I never really thought of Canada as a "foreign" country.   I should have used PayPal to avoid the transaction fee. 

     
"This has not only been fun, it's been a major expense." - Gallagher

MRTT

Obi110.  I have Google Voice setup and working on sp1, nothing on sp2.   Nothing on line.

Caller ID number appears fine on my phones.
I would like to have it go to my syslog too...

I have my obi setup to send syslog info to my syslogger...
I have 'X_SipDebugOption' parameter set to 'Log All Messages.'

I have tried all the various syslog levels available.  I infer level 7 should work.  However when calls come in, there is no corresponding syslog info arriving.

I do see some debug info when the OBI110 reboots, like:
SLIC & DAA is initialized
Start Main Service Now
PROV:Auto Update invalid server name()

So what am I missing?

QBZappy

Quote from: MRTT on October 21, 2011, 08:40:06 PM
Obi110.  I have Google Voice setup and working on sp1, nothing on sp2.   Nothing on line.

Caller ID number appears fine on my phones.
I would like to have it go to my syslog too...

So what am I missing?

It's been requested a few times before. Obihai has not made the syslog more verbose for some reason. You would expect this to be a relatively easy feature request to provide. What you see in the syslog is what you get. A lot of 3rd party apps/utilities could feed off the syslog output.
Owner of the 1st OBi110/100 units in service in Canada & South America. 1st OBi202 on my street. 1st OBi1032 in Montreal.

infin8loop

Ok, now I'm confused.  This isn't the first time I've seen mention in different threads about not seing callerid number in the syslog.  Scroll back in this thread to message http://www.obitalk.com/forum/index.php?topic=1723.msg11147#msg11147  and download it's attached file named obisyslogd-commented.txt.  I added comments to this syslog starting with ****COMMENT to document each incoming call type.   Look for
[2011.10.19 01:54:07] [192.123.1.12] <7> [SLIC] CID to deliver: '' 11231234567   This is a message level 7 from an incoming call over Google Voice.  The message data is "raw" straight from the OBi beginning with the <7>.  The [date time] and [obi ip iddress] prefix is added by the syslog script.  The 11231234567 is a masked callerid number in North America format with a "1" followed by 10 digit phone number (The actual number was correct before I manually masked it).  The empty '' denotes that no callerid name was present. I consistently see this message on incoming Google Voice calls.    'X_SipDebugOption'  does not have to be enabled but it could be.  This syslog was obtained from my OBi110 with X_SipDebugOption=Disable.  I get the same basic results no matter if I run my perl syslog script which created this syslog file or the standard syslogd package I installed in Ubuntu linux.  The OBi110 is configured with the default Syslog Level 7 (default check box checked).  Firmware 1.3.0 (Build: 2586), but it's been working this way for the last couple or so versions at least.  I currently have google voice on SP1 and voip.ms on SP2.  But it worked the same way with just google voice on SP1 and nothing on SP2. What am I missing??!  I doubt it's my mystical programming powers that are making the difference (heh).


"This has not only been fun, it's been a major expense." - Gallagher

etc6849

infin8loop, I don't think you're doing anything wrong.  This is what I'm getting with the latest firmware.  Unfortunately, I don't have an older firmware to try.

QBZappy and MRTT: I'm not sure why Google Voice caller id numbers do not show.  I'm using GV integrated with Sprint cellular service, so maybe that's why the number is working for me?  I do know I'm running the latest firmware.  I also know they show up when I use Kiwi Syslog.

I've enhance my perl script to take the syslog packet and look for ''.  Now if it sees '', it will look up the name using http://freecnam.org/dip?q=8037320011 

I figured out the problem with freecnam.org was that their site was not working the day I was trying it.  Briefly this morning it wasn't working.  I'm not complaining though, it's a very handy site.

I'll post the script after I finish things and test them more.  The idea is to first try freecnam if no caller name is present in the packet; if it doesn't return anything, look up the caller's city and state using http://www.telcodata.us/query/queryexchangexml.html?npa=316&nxx=265.

I appreciate all of the help you guys have given me.  I've leanred a lot already!

QBZappy

Quote from: oleg on March 27, 2011, 08:46:17 AM
I am curious why nobody mentioned SysLog facility. It's already implemented in OBi110 and for SIP calls can send more than enough information to the destination computer (although for GV calls it lacks call details - I hope OBi developers will catch up here  ;)). Quite simple processing script may extract necessary details and pop-up message. Just my $0.02  :)

Emphasis added

Back in the day the syslog seems to have been less verbose. It looks like they quitely improved the syslog output.
Owner of the 1st OBi110/100 units in service in Canada & South America. 1st OBi202 on my street. 1st OBi1032 in Montreal.

PeterFales

I'm puzzled by missing syslog information too.  A few months ago (November 2011) I wrote a perl script to watch for the "[SLIC] CID to deliver" messages and it was working fine.  Today when I went to test it again, I'm not getting those messages.   I get other syslog <7> messages from the OBi, so I think I have everything configured correctly.   Did something change in my OBi configuration?  a firwmare update?  the google-voice protocol?  or ???

etc6849

See this post: http://www.obitalk.com/forum/index.php?topic=2368.0

The firmware has changed and they removed some data from the syslog output.  No more outgoing call log :(

w84no1

I have setup a syslog server in perl to monitior the Obi and send caller id info to each of my Windows  Media Center PCs.  I use GV so I have no caller id info except the phone number.  Here is the code I used, which includes the number lookup from freecnam.org.  Enjoy.


use IO::Socket::INET;
use LWP::Simple;
use subs::parallel;
use XML::Simple;
use Data::Dumper;
use feature "switch";

#start writing syslog data to client
my $syslog = IO::Socket::INET->new(Proto => 'udp', LocalPort => '514');
die "Could not create syslog socket: $!\n" unless $syslog;
print "Syslog server is online.\n";

sub formatPhoneNumber {
my $number = shift;
die "$number is not a valid 10-digit phone number" unless $number =~ /^\(?([1-9]\d{2})\)?[-., ]*?([1-9]\d{2})[-., ]*?(\d{4})$/;
my $formatted = "($1) $2-$3";
return $formatted;
}

sub connectMCE : Parallel {
my $PeerHost = shift;
my $PeerPort = shift;
my $CID = shift;

#start writing data to client
my $mce = IO::Socket::INET->new(Proto => 'tcp', PeerHost => $PeerHost, PeerPort => $PeerPort);
if($mce) {
print "Connected to MCE $PeerHost on port $PeerPort.\n";

$data = <$mce>;

# write on the socket to server.
$data = 'msgboxrich "Phone Call" "' . $CID . '" 20 "OK" "nonmodal" "c:\phone.png"';
print $mce "$data\n";
print "$data\n;"
}
else
{
print "Could Not Connect to $PeerHost on port $PeerPort.\n";
}
close($mce);
}

#if $syslog is defined, then continuously monitor the socket for incoming data
while (defined($syslog))
{
recv($syslog,my $msg,1500,0);

# clean up msg some
# if msg = "<7> [SLIC] CID to deliver:" then send caller id to mediacenter
if ($msg =~ /CID/) {
$msg = substr($msg, -12);
$msg = substr($msg, 0, -2);
print $msg . "\n";
my $content;

                #todo - lookup number is address book
given($msg) {
when (/9999999999/) { #removed real phone number
$content = "DAVID'S CELL";
}
when (/8888888888/) { #removed real phone number
$content = "DAVID'S OFFICE";
}
default {
my $url = "http://freecnam.org/dip?q=" . $msg;
$content = get $url;
print "$url\n"
}
}

my $fullCallerIDInfo = formatPhoneNumber($msg) . " - " . $content;
print "$fullCallerIDInfo\n";

# create object
$xml = new XML::Simple (KeyAttr=>[]);

# read XML file
$data = $xml->XMLin("data.xml");

# dereference hash ref
# access <client> array
foreach $e (@{$data->{client}})
{
connectMCE($e->{peerName}, $e->{peerPort}, $fullCallerIDInfo);
}
}
}
close($syslog);