Obi/Google Voice disconnecting overnight

[janice]

Dear SteveInWA, so am I to understand that the fact i upgraded the firmware hours prior to this happening, was concidental -- then running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

I ask because if I understand you correctly i would retract my post prior to this and feel a lot better. thanks for your time.

[vtsnaab]

Thank You Mango, for being so kindly helpful.

Yes, you can use ***01#.
Alternately, the SoftwareVersion (same thing) is displayed on the device's internal web server under Status >> System Status.
If the problem returns, try to dial ***6, 1.  If that doesn't work, follow the instructions at http://www.obitalk.com/forum/index.php?topic=11256.msg73954#msg73954 .  Let us know how things go.

As of later that day my Obi110 came back to normal and is fine.

My .02 in this matter is that essentially I agree with the poster named Janice - these folks have taken a very poor stance and show extreme disregard for their users IMO.

Using a forum to (mostly) get support from other users rather than having any direct support from the parent company is OK and not unusual, BUT:
Making users pay via indirect means and bricking devices are grand methods to make enemies IMO.

Also - like Janice, I am not averse to paying a small yearly fee; compared with a monthly landline bill it would be trivial - and if it became a large fee there are other ITSPs who offer very fair pricing.

The other spinoff of all this forum usage is that -IF- Obihai had properly scouted their territory as time went on, they would have realized that their users would find out about all the permutations of connecting with the world via VOIP far more quickly via exposure to other users than they would have if there had only been direct company support.

An ancient tale from me which relates:
Once upon a time I had a friend who was a real, geniune software genius.
He made a product that was unique and way ahead of its time.
It sold well and he made plenty from the sale of it.
Then the troubles began - here's why:
He was too cheap to create any support options from his own resources AND he despised doing it himself simply because he wanted to be writing code and NOT answering calls.

The moral of this story ??:
Bankruptcy, as his users got discouraged and simply turned their backs, no longer spreading word of his great software.

The customer is ALWAYS boss (as the old saying goes) and can easily fire everyone in any company right up to the owner - this is neither new nor is it likely to change.

[Mango]

But nice people -- please have a great weekend and thanks again for getting me up and running until I find a replacement for this company.

There are plenty of great service providers besides Google Voice that are compatible with your OBi ATA.  I see you're already familiar with Callcentric.  As well, popular service providers include VoIP.ms and Anveo.  If you don't want to use Google Voice or OBiTALK, I understand, but I urge you not to abandon Obihai's hardware.  Their ATAs truly are the best on the market today.

running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

I would not assume Google caused the problem, unless there is information suggesting that, that I'm not aware of.

Regardless of who caused the problem, Obihai certainly responded to it poorly.  They could have pushed the firmware update automatically, or at the very least updated OBi-latest.fw and made ***6 operational.  (Some users reported ***6 worked and some reported it did not.)  I don't have a way of testing ***6, but OBi-latest.fw is still build 2872.

[Taoman]

Dear SteveInWA, so am I to understand that the fact i upgraded the firmware hours prior to this happening, was concidental -- then running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

Steve is more trusting of Obihai than I am. It is my belief this whole incident happened because Obihai released a new version of firmware for the 1xx series. Those people with an OBi1xx series device out of warranty and also having a Google Voice configuration began having issues. I have an OBi110 but do not have a Google Voice configuration and I had no issues at all with my device even with the older firmware. A similar situation happened when a new version of the 2xx series firmware was released although it wasn't as severe as this one apparently was. When the new 20x series firmware was released it seemed to only affect those who were trying to change their password or GV configuration and their devices were out of warranty. It didn't actually make the devices stop working if I remember right.

I am not saying Obihai planned for this to happen in the manner it did. I have no proof of that. But I am saying there was no "fix" released by Obihai. Google did not suddenly change their authentication protocol and Obihai had to quickly fix something. If that were the case why didn't it affect any 20x devices and why did this incident only happen to 1xx devices that were out of warranty?

What happened is Obihai released a new version of firmware for the 1xx series and it inadvertently, or on purpose, caused a lot of problems for a lot of people. The only fix was to update to that new firmware.

[SteveInWA]

Dear SteveInWA, so am I to understand that the fact i upgraded the firmware hours prior to this happening, was concidental -- then running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

I ask because if I understand you correctly i would retract my post prior to this and feel a lot better. thanks for your time.

It's likely coincidental that your prior update was the root cause of your problem.  More likely, your update just happened to coincide with this outage, and by rebooting the device, you caused it to go into the failure condition.

Google didn't change it's authentication method "out of nowhere".  This change was made more than 2 years ago.  The new authentication method is more secure, and involves generating and exchanging secure access tokens, stored on your device, to authorize access to the Google Chat service for your Google account.  Obihai released compliant firmware at that time, to allow all of its devices to continue to work with Google Chat.

An undisclosed, latent issue with the build 2872 firmware for 100/110 devices caused a failure to authenticate, starting the other day, with the OBiTALK portal's code that updates the keys.  Obihai updated the firmware to the 2886 level, to fix this bug.  This is the first such issue with the OBi/Google Voice authentication system since the new authentication method was introduced two years ago, and I don't expect it to be a regular occurrence.  

Again, while this was poorly handled from a PR standpoint, they did promptly jump on the problem and fix it.

[Taoman]

An undisclosed, latent issue with the build 2872 firmware for 100/110 devices caused a failure to authenticate, starting the other day, with the OBiTALK portal's code that updates the keys.  Obihai updated the firmware to the 2886 level, to fix this bug.  This is the first such issue with the OBi/Google Voice authentication system since the new authentication method was introduced two years ago, and I don't expect it to be a regular occurrence.  

Is that conjecture on your part or did Obihai actually directly inform you of this?

[SteveInWA]

Dear SteveInWA, so am I to understand that the fact i upgraded the firmware hours prior to this happening, was concidental -- then running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

What happened is Obihai released a new version of firmware for the 1xx series and it inadvertently, or on purpose, caused a lot of problems for a lot of people. The only fix was to update to that new firmware.

I am not aware of any new firmware for the 100/110 series being introduced recently (I don't recall how long ago 2872 was released, but I believe it's been the one and only, stable release for a long time, having been released to update to the OAUTH 2.0 authentication method.

What I think happened, is that the 2872 firmware had some sort of date bug in it, that caused it to fail to update its keys.  The 2886 firmware released on the day you found it fixed that bug.

This certainly wouldn't have been a "time bomb" intentionally placed in the code; even the dumbest company wouldn't do that, knowing how today's social media and online forums would send the company into the ditch for such a practice.  As you pointed out, there was no corresponding bug in the 200 series firmware.  While the warranty status does muddy up and confuse the user when it comes to obtaining firmware upgrades, as long as a firmware upgrade is available, there is nothing preventing a user from manually upgrading their device.  In this particular instance, my guess is that the date in the device certificate stored on the 100/110 products improperly triggered some setting in Obihai's entitlement system, and thus locked those devices out configuring services via the portal.  Again, Obihai promptly fixed that issue after I reached out.

[SteveInWA]

An undisclosed, latent issue with the build 2872 firmware for 100/110 devices caused a failure to authenticate, starting the other day, with the OBiTALK portal's code that updates the keys.  Obihai updated the firmware to the 2886 level, to fix this bug.  This is the first such issue with the OBi/Google Voice authentication system since the new authentication method was introduced two years ago, and I don't expect it to be a regular occurrence.  

Is that conjecture on your part or did Obihai actually directly inform you of this?

I contacted Obihai to let them know this was happening.  They replied that they'd get right on it.  Soon after, you found the 2886 firmware.  They never got back to me directly, nor, of course, did they post anything here.

The fact that they needed to (and were able to, without much analysis) update the firmware, and, knowing what I know about Google Voice and OAUTH 2.0, and how Obihai installs digital certificates on its devices, I'd call it an "educated guess".

[Taoman]

I contacted Obihai to let them know this was happening.  They replied that they'd get right on it.  Soon after, you found the 2886 firmware. 

But the firmware was released before that time. You posted at 2:47 on the 18th you were going to see what you could find out. I can only assume it was sometime after that point that you "reached out" to Obihai.

Rchandra posted he had already paid the $10 and got the updated firmware at 2:22 on the 18th. The updated firmware (not a fix) had already been released by the time you contacted Obihai.

If you truly believe your story, Steve, you need to answer these questions:

1>Why didn't this authentication problem affect 20x devices?
2>Why did this authentication problem only affect 1xx devices that were out of warranty?
3>Why did this problem suddenly crop up after all this time and only on 1xx devices?

The answers you've come up with so far stretch credulity.

[SteveInWA]

I contacted Obihai to let them know this was happening.  They replied that they'd get right on it.  Soon after, you found the 2886 firmware.  

But the firmware was released before that time. You posted at 2:47 on the 18th you were going to see what you could find out. I can only assume it was sometime after that point that you "reached out" to Obihai.

Rchandra posted he had already paid the $10 and got the updated firmware at 2:22 on the 18th. The updated firmware (not a fix) had already been released by the time you contacted Obihai.

If you truly believe your story, Steve, you need to answer these questions:

1>Why didn't this authentication problem affect 20x devices?
2>Why did this authentication problem only affect 1xx devices that were out of warranty?
3>Why did this problem suddenly crop up after all this time and only on 1xx devices?

The answers you've come up with so far stretch credulity.

I'm not on the witness stand. I am not going to argue timelines.  For all I know, Obihai already found and fixed the issue by the time I contacted them; as we know, their communications skills are abysmal.  We also know that the various firmware update methods often are out of sync (***6 vs. downloading the supposedly "latest" generic firmware link, vs. downloading a build-specific firmware link).  It's terrible version control, and users shouldn't have to do detective work to find the truly latest firmware.

My own experience was that the firmware update was required to fix the issue; there is no reliable (multiple user reported) evidence that the 2872 build could be made to work.  Personally, I took a first-generation, out-of-warranty 110 device out of storage and plugged it in.  It was already running 2872.  It was configured with two different valid/working Google Voice accounts, one on SP1 and the other on SP2.  SP1's Google account had an expired password, which I successfully updated.  SP2 worked out of the box.  So, both then worked briefly that evening, then both SPs failed the next morning, consistent with symptoms reported by others.  After updating to 2886, it worked.  That's all that matters to me.

The 100 series devices have an entirely different SoC (MIPS-based) and thus, different firmware, than the ARM-based 200 and 10x2 series devices.  I think it is reasonable to assume that Obihai made a mistake in either the 100-series device firmware or back-end systems that handle security for those particular devices.  Digital certificates have expiration dates.  OAUTH keys are periodically refreshed.  Something broke that procedure.  They fixed it.

This is the most plausible explanation I can offer, and yes, it is only my educated guess.  If you feel it's valuable and productive to come up with a different educated guess, you're welcome to it.

Obihai's treatment of firmware upgrades on in-warranty vs. out-of-warranty devices has always been vague, inconsistent, and poorly-implemented.  I don't think it's productive to try to reverse-engineer what happened with regard to a device being in-warranty or not.  As we've seen, out-of-warranty devices can be updated manually, and, this case was no different; as soon as the new firmware was released, everyone was able to update their devices, regardless of warranty status.

I have no reason to believe that there was any malice or conspiracy here; just a technical error.  If there had been some evil-doings, then a) the 100/110 devices would now be permanently bricked, which is most certainly not the case, and b) other evil things would have happened, such as bricking out-of-warranty 200 and 10x2 devices, which again, didn't happen.  Haven't you ever made some sort of error in your work?  Software companies discover f-ups every day, the hard way, in their code, when things like this happen.

Bottom line:  old news about Obihai's misleading and distracting "advertisement" to buy the $10 support upgrade, but no change in the ability to manually update the devices, once a newer build becomes available.

Several of us have directly complained to Obihai about this messy situation.  I've personally brought it up with Sherman.  I have no way, short of a Carl Ichan-style takeover, of controlling their behavior.

[Taoman]

I'm not on the witness stand.

Didn't intend for you to feel like you were being cross examined. Obviously I draw a different conclusion from the available evidence than you do. It would be interesting to read the Release Notes for this firmware version if Obihai were ever to post them.

[LTN1]

I'd like to cross-examine the witness.

Why cha you do it Steve? Why?



Where am I?

[janice]

Thank you everyone for your responses. I really appreciate it. There are several facts i can add to this discussion which might be of interest.
One: I habitually log into my Obi about every month or so at the most. This was a software update waiting that was not there prior to my last login, which i believe was actually only last week. Two: Last night, before my latest post on this board I tried dialing **x(whatever) someone said to do and it said "software unavailable". Three: I then downloaded Taoman's contributed link to the firmware upgrade and all of a sudden everything worked again. Auto firmware update has been disabled for years. The uptime on the obi box was about 10 days -- it had NOT been running continuously without upgrades when it went out of warrantee in 2013.

I provide this fact pattern b/c I want to be clear I also feel it was connected to my upgrade. Everything happened an a 30 hour period or so after that.

I appreciate Mango's point that this is a great device. And in fact I DO maintain a Callcentric account, as well as a landline. I work 4-5 hours a day on the phone.

But my point is my impression of Obi as a company has gone from a 10 to a 1.5 and is unlikely to recover because they have treated their happy customer base (for instance me just buying an extra one and putting it on a shelf!) with either a very stupid lazy aggravating manipulation to get money -- OR really really terrible QC of a firmware upgrade and in both cases *ZERO* communication. They can send me product announcements, but they can't send an email about paying $10?

That ain't right. Truly, there is no assurance that when we are next to log in to our devices - that Callcentric will not work without a $10 payment b/c of an unspecified security patch we cannot download without making a "contribution".

As I said earlier, I got no problem paying, but I do have a problem with a thug sticking gun in my back when I am in a great mood whistling just walking down the street.

And now the Obi200 in the box. When I open for the first time, it will it be out of warrantee? Stay tuned.

Thank you all SO much for your insights and input. This has been interesting and you all are very kind supporting this user base and folks like me.
-janice

Dear SteveInWA, so am I to understand that the fact i upgraded the firmware hours prior to this happening, was concidental -- then running into all these issues was b/c google changed it's authenticaton methods out of nowhere, and then the firmware posted by Taosman (sp) which repaired the current problem was Obi responding to the problem?

Steve is more trusting of Obihai than I am. It is my belief this whole incident happened because Obihai released a new version of firmware for the 1xx series. Those people with an OBi1xx series device out of warranty and also having a Google Voice configuration began having issues. I have an OBi110 but do not have a Google Voice configuration and I had no issues at all with my device even with the older firmware. A similar situation happened when a new version of the 2xx series firmware was released although it wasn't as severe as this one apparently was. When the new 20x series firmware was released it seemed to only affect those who were trying to change their password or GV configuration and their devices were out of warranty. It didn't actually make the devices stop working if I remember right.

I am not saying Obihai planned for this to happen in the manner it did. I have no proof of that. But I am saying there was no "fix" released by Obihai. Google did not suddenly change their authentication protocol and Obihai had to quickly fix something. If that were the case why didn't it affect any 20x devices and why did this incident only happen to 1xx devices that were out of warranty?

What happened is Obihai released a new version of firmware for the 1xx series and it inadvertently, or on purpose, caused a lot of problems for a lot of people. The only fix was to update to that new firmware.

[Mango]

Truly, there is no assurance that when we are next to log in to our devices - that Callcentric will not work without a $10 payment b/c of an unspecified security patch we cannot download without making a "contribution".

If you disable OBiTALK Service, and only manually install firmware updates after they've been out a while, it is highly unlikely that the device would suddenly cease to function with Callcentric (barring outside issues like physical damage).  Nothing is impossible of course, but every widespread issue with OBi ATAs that I can think of has been caused by a firmware upgrade.  The SIP-based service providers don't require constant firmware upgrades.  Google Voice is a different animal due to it using a different protocol and authentication method.

[janice]

Got it. And again, thanks for the education and the info. I didn't know - or at least think through about the engineering side of it.
-janice

If you disable OBiTALK Service, and only manually install firmware updates after they've been out a while, it is highly unlikely that the device would suddenly cease to function with Callcentric (barring outside issues like physical damage).  Nothing is impossible of course, but every widespread issue with OBi ATAs that I can think of has been caused by a firmware upgrade.  The SIP-based service providers don't require constant firmware upgrades.  Google Voice is a different animal due to it using a different protocol and authentication method.
[/quote]

[pvkid]

Download the following new firmware version and install manually:

http://fw.obihai.com/OBi110-1-3-0-2886.fw

Please report back if that fixed the problem.

I followed these directions and upgraded with no problem. Anyone have any ides how to update a remote device that is still connected to the network?

the pvkid

[vtsnaab]

Looks like they figured out a way to really tighten the thumbscrews on folks with older Obi boxes now - as mine utterly refused to re-authorize today (right after getting a call).

Fortunately I was able to update it and all is well - for now - but these folks are NOT making any friends by continuing down their chosen path.

[jmbrowning]

I followed these directions and upgraded with no problem. Anyone have any ides how to update a remote device that is still connected to the network?

the pvkid

Depends on how good the network connection is for the remote obi host.

I have tried ssh'ing into the router via satellite internet and from there running a tunnel to port 80 on the obi box. I have tried to access the call history but I found the satellite internet connection to be very janky. Sometimes it works and sometimes it hangs. I personally would not feel comfortable uploading a firmware update that way. It might be more reliable over a landline connection (DSL/cable/fiber/etc) and a SSH http tunnel to the obi host might be more feasible.

If the connection to the remote network is unreliable, it might be better to actually ssh into another host on the network and download the firmware there and see if you can do the update through a console browser like lynx. You could try running gnu screen so that if you get disconnected the update can continue.

Hope that helps

[ChuckInNH]

Yikes.  I went to make a call a couple hours ago only to be greeted with the voice message that I needed to reauthenticate Google Voice because of a password change, which wasn't correct, since nothing changed.  The Obitalk web site had disabled the Settings button so it would not let me attempt this.  I tried deleting the SP1 (which it let me do) only to discover it would not allow me to run Setup again, so SP1 was now not configured to anything.  And, I saw the ransom note at the top about the device being out of warranty... since February BTW and it had been working great so what's special about now I have no idea.   Yeah... no way am I paying that with the hope that it might fix the problem.  I also tried ***6 but it said no updates were available.

Thanks to this forum and DSLReports, the link posted to the firmware worked.  It works again.  But, I am extremely unimpressed.  If Obihai wishes to charge an additional fee for some kind of additional service on top of the base services that I paid for, I'm happy with that and would even consider paying depending on what it is.  But, bricking the device with no warning, giving me no useful info about what is actually wrong, not sending me any email about it or anything, and appearing to require sudden payment to make it all work again... well, I am very unpleased and will not be buying any more Obihai products in the future nor recommending them to anyone.

[SteveInWA]

Yikes.  I went to make a call a couple hours ago only to be greeted with the voice message that I needed to reauthenticate Google Voice because of a password change, which wasn't correct, since nothing changed.  The Obitalk web site had disabled the Settings button so it would not let me attempt this.  I tried deleting the SP1 (which it let me do) only to discover it would not allow me to run Setup again, so SP1 was now not configured to anything.  And, I saw the ransom note at the top about the device being out of warranty... since February BTW and it had been working great so what's special about now I have no idea.   Yeah... no way am I paying that with the hope that it might fix the problem.  I also tried ***6 but it said no updates were available.

Thanks to this forum and DSLReports, the link posted to the firmware worked.  It works again.  But, I am extremely unimpressed.  If Obihai wishes to charge an additional fee for some kind of additional service on top of the base services that I paid for, I'm happy with that and would even consider paying depending on what it is.  But, bricking the device with no warning, giving me no useful info about what is actually wrong, not sending me any email about it or anything, and appearing to require sudden payment to make it all work again... well, I am very unpleased and will not be buying any more Obihai products in the future nor recommending them to anyone.

Obihai fixed the issue immediately.  Yes, their communication skills blow, but you were able to get it working, for free.