We had fewer hands and less time. The real problems were RAN-related and infrastructure-related, mostly due to poor management of the preparations on my end.  I had a lot of unexpected distractions in July that screwed my schedule.  I apologize.

Power was a serious problem.  We were offline a lot. The inverter had been reliable for 3 or 4 previous years, but it totally screwed us this year. Not only could we not rely on it to run the system overnight,  but the fast power cycling left us with a corrupt boot sector on the Yate server that probably cost us half a day of operation and may even have lead to the registry bug (a typo in the Javascript) that had us down for part of Monday.

On the list of improvements over last year, though, the Yate-based registry supported a much higher throughput than the old sipauthserve, which had been a bottleneck in previous years.  Although it took a couple of days to get adequate SMS throughput, the overall Yate framework of MySQL + Javascript was a very good one and has a lot of long-term potential.

On Sunday, the main problem with SMS delivery was not that there were not enough delivery attempts, but that so few of the delivery attempts were successful, probably because by Sunday afternoon, 2 of our 4 BTS units were offline and the remaining 2 had shifted into automatic power reduction to try to protect themselves from the high registration loads.  Having fewer tower sites didn't just mean less coverage; it also meant less total RAN capacity, which concentrated the registration loads into fewer channels and meant less excess capacity to absorb a BTS failure.

Should we do it again?  Yes, but only if we can stage properly at least two months ahead of time.  This could be done in our SF test network.  Coordination with the City of SF has been slow, but is finally moving.

On Sep 10, 2013, at 7:49 AM, Johnny Diggz <jhiggins@tropo.com> wrote:

It is significant to note, we had significantly fewer technical hands than we did last year.   Our fewer towers was not lack of effort, rather time.

On Tue, Sep 10, 2013 at 6:52 AM, tim panton <thp@westhawk.co.uk> wrote:

Chief Evangelist, Tropo
Tropo.com || Geeks Without Bounds || Diggz.org
@johnnydiggz || diggz@tropo.com
How ServiceNow helps IT people transform IT departments:
1. Consolidate legacy IT systems to a single system of record for IT
2. Standardize and globalize service processes across IT
3. Implement zero-touch automation to replace manual, redundant tasks
Openbts-bm2009 mailing list