[ALU]: "signalasis" Undermining Customer Experience and Operator’s Reputation

An article by Josee Loudiadis [pictured], Director of Network Intelligence, Alcatel-Lucent, covers the "signaling spikes that can overwhelm available signaling capacity, undermining the customer experience and the operator’s reputation with it" and analyzes the signaling behavior of several common applications.

"There are three kinds of “signalasis”: microbursts that can be measured in seconds, extended bursts that can last minutes to hours, and suddenly sustained signaling growth where signaling jumps significantly at one point in time and continues to increase over weeks and months. Facebook’s 60% jump on Nov 2012 is an example of the latter" 

• Extended bursts Viber outage: 

GGSN experiencing 92% jump in signaling
 for 4 ½ hours during Viber outage

On April 29th, CPU overload alarms reported that RNCs (Radio Network Control) were inundated with requests. The signaling spike could be matched to Viber – flows showed that Viber servers were no longer responding. Viber was down. But, why would this app outage have such a significant impact on signaling resources? The answer is in Viber’s call failure handling: the app would retry repeatedly to connect with the server, creating a larger signaling wave as more users tried unsuccessfully and repeatedly to connect. 

• Microbursts:   Microsoft Exchange and iOS: 

GWs subjected to 36% signaling microburst at midnight
due to iOS-based Microsoft Exchange

This case exemplifies other short term outages where the signaling spike exceeded the signaling capacity on a daily basis. A 36% signaling jump occurred everyday at midnight, but the reason for the spike remained mysterious. The WNG narrowed it down: the signaling was initiated by devices trying to reach the Microsoft Exchange server. This interaction lasted less than 1 second in duration. It only involved iPhone devices and was more predominant with iOS version 6.1. Equipped with this information, operators could contact Apple and identify the root cause. A fix was issue in a subsequent iOS version update

See more  - "Anatomy of signaling spikes: Viber, Microsoft Exchange & a pre-loaded app" - here.

ALU Identifies the Chatty Mobile Apps

A new report by Alcatel-Lucent ",, examines the impact of leading mobile applications on service provider networks and on consumers’ data plan and device battery life .. The rankings in this report are derived from mobile network analytics provided by the Alcatel-Lucent Wireless Network Guardian [see "Softbank [Japan] Uses ALU for Better QoE" - here]. All measurements are based on actual app usage by more than 15 million subscribers on live 3G mobile networks in North America, APAC and the Middle East"

Key findings:

• Apps that impact the network due to their high data volume — a group that includes most video apps, along with iTunes®, Instagram, Pinterest, Apple Maps and Pandora — are targets for bandwidth optimization. One service provider reduced cell utilization by an average of 20% after introducing selective traffic optimization.
   
• The apps with the biggest impact on the consumer data plan are YouTube, Netflix, Facebook Video, Pandora and Video on Instagram.
   
• The apps in the High Signaling quadrant are perfect candidates for code design optimization aimed at reducing the number of times the device and network must establish communications
  
  
  

Bandwidth optimization success story: Selective traffic optimization

A service provider knew that the key to managing traffic cost was to manage the peak hour. The peak hour drives CAPEX investment, so logic dictated that reducing the load at that critical time would reduce the need for investment. Studies of  peak-hour traffic revealed a surprising fact: that the typical heavy users were not active during this period. Each day, a different set of users topped the peak-hour volume chart. Throttling certain users during a certain period of the day would not help the service provider reduce utilization.

The service provider selected a two-part mechanism to optimize traffic. The first part involved using network analytics (specifically, an algorithm relying on utilization and subscriber QoE metrics) to determine when congestion occurred, relay the severity of the congestion, provide a list of subscribers currently attached to the cell, and rank these subscribers by data volume. The second component was to use a policy manager to decide whether specific users should be throttled, how many users should be throttled and for what period of time they should be throttled. The service provider reduced cell bandwidth  by an average of 20% and reduced the amount of time cells spent in congested state each day by 60%.

See "Alcatel-Lucent Mobile Application Ranking Report" (registration required) - here

NSN: VoLTE Beats OTT VoIP 4:0 !

Gerald Reddig [pictured], Nokia Solutions and Networks, explains in company's blog why VoLTE is good for MNOs - technically.

"NSN Smart Labs recently tested both client groups, OTT VoIP and VoLTE" and found that in all 4 measured KPIs VoLTE is better:

• VoLTE clients consumed approximately 40% less than OTT VoIP clients.
   
• OTT VoIP clients generated up to 10x more data connections in the mobile network than a VoLTE client during a call
   
• Most OTT VoIP applications required between 20% and 40% more throughput than VoLTE clients during active calls to achieve this quality level, although one OTT VoIP application showed exceptional performance and came close to VoLTE. Overall data volume consumed over a period of time, including a mix of active and stand-by periods, resulted in at least 50% lower consumption for VoLTE thanks to its more efficient behavior during stand-by
   
• Typical default OTT VoIP keep-alive patterns activated during standby created between 100% and 200% higher signaling load on LTE networks compared with VoLTE

See "Why operator VoLTE beats OTT VoIP" - here.

NSN: How can MNOs Deal w/OTT Resource Consumption and Lost Revenues?

A post by Andrew Burell [pictured], Nokia Solutions and Networks, to the company blog says that "OTT is certainly not a lost cause to MNOs".

" .. [MNOs] see OTT applications such as Skype, WhatsApp and NetFlix piggy-backing on their network investments, clogging up resources and stealing their revenues. The figures themselves are certainly frightening. Every day 2,371 new applications are born, and a single ‘rogue’ app can generate 2,422 signaling messages per hour. By 2017, there will be 32 trillion messages sent by OTT messaging applications, and by 2020, operators stand to lose a cumulative total of $479 billion to OTT players.

.. With the right tariff structure for data-services in place, there are a number of different strategies that operators can successfully adopt to turn threat into opportunity. For example:

• Partnering with OTTs: ‘3’ offers subscribers a ‘Roaming Pass’ that allows them to use Whatsapp abroad with an inclusive data allowance
   
• Operators can launch proprietary services of their own: Verizon has followed this approach with its ‘Viewdini’ service – a mobile video aggregation portal that delivers mobile content from Netflix, Hulu and others.
   
• .. [NSN] Smart Labs and NPO (Network Planning and Optimization) expertise with performance proof points such as:
    
• Smart Labs research has shown that activation of WCDMA software suite features like Fast Dormancy Profiling can reduce signaling loads by 40%  [see "NSN: New Radio Software Reduces WCDMA Network Signaling" - here]
   
• Our NPO video optimization service was able to reduce content buffering by 30% for a major European operator

See "OTT – Horror movie or romantic comedy?" - here.

NSN Warns MNOs: OTT Outages Increases Signaling (20X!)

A blog post by Gerald Reddig [pictured], Head of Smart Labs Marketing, Nokia Solutions and Networks, reports that "Recent OTT service outages in cloud service provided by Google hit mobile operator radio access networks with an unexpectedly high signaling load. Online services rely entirely on the service availability in large data centers provided by Google, Amazon and Microsoft, for example.

NSN Smart Labs verified that a cloud service outage can trigger smart phones to generate signaling loads up to 20 times greater than normal. This is an important heads up for mobile operators to prevent chain reactions that could lead to network outages .. For the test, NSN Smart Labs utilized one popular high-end Android smart phone. Now multiply that signaling storm by thousands of smart phones simultaneously and radio access networks would definitely be compromised".

See "OTT service blackouts trigger signaling overload in mobile networks" - here.

NSN: New Radio Software Reduces WCDMA Network Signaling

 
Nokia Siemens Networks launched "..new software applications for the Liquid Radio WCDMA Software Suite to help mobile broadband manage the smartphone boom and substantially reduce the network signaling overload .. 

• Fast Dormancy Profiling is a new Nokia Siemens Networks technology that identifies the legacy smartphones connecting to the network and eliminates the unnecessary signaling they often create. This application significantly reduces the air interface load on the Radio Network Controller (RNC). [see also "[NSN]: Chatty Android Phones should Move on to Network Controlled Fast Dormancy" - here]

• With Fast Cell PCH Switching, also a Nokia Siemens Networks technology, the network learns how specific smartphones are using the network services and shortcuts the signaling process for rapid call set-up, faster response from mobile broadband services and reduced signaling.

• High Speed Cell FACH enables networks to support more smartphones and provide faster services for subscribers using lower data volumes. It also significantly decreases signaling load between the RNC, base station and smartphone.

See "Liquid Radio WCDMA Software Suite helps operators reduce signaling, manage the smartphone traffic boom" - here.

ALU: Signaling Issue with New Facebook

Lindsay Newell, VP, Marketing Networks & Platforms Group, Alcatel-Lucent, writes in a post to ALU's blog that "On November 15, Facebook released new versions of their mobile app for Android and iOS. Prior to the new release, Facebook signaling and airtime already accounted for 10% and 15% of the overall load on 2G/3G networks, respectively. As users around the world updated and started to use this new version, we quickly noticed a dramatic increase of almost 60% in the signaling load and 25% in the airtime consumed by the Facebook application. During the same period, the number of Facebook users increased by only 4%. Clearly, it is not the swelling of Facebook’s community that intensified the load, but rather the introduction of new Facebook features for mobile users and underlying platform changes .. The effect on service provider networks was significant, driving up overall (total) signaling traffic and airtime consumption by 5-10%".

"Another trend we detected was an increase in popularity of Facebook video traffic — a whopping 350% increase in volume since the November 15th launch — the most growth of all Facebook traffic".

See "New Facebook: not only draining your personal time, but mobile network capacity as well" - here.

Telus Sees 2,700% Signaling Growth Y/Y

 
Perry Hoffman reports to Cartt.ca on the "'tsunami of traffic' resulting from very high levels of social media over wireless devices and the connectivity of an increasingly large number of those devices is forcing mobile operators to deal with growing congestion. This was one of the key messages from Eros Spadotto (pictured), executive vice-president of technology and operations at Telus during a keynote presentation to the Institute of Electrical and Electronics Engineers (IEEE) InternationalConference on Communications on Tuesday in Ottawa".
 
"The reality is that while our megabytes of traffic is increasing, the signaling from these devices is greatly overwhelming that. In fact, we can look at a period of time where our growth has been 100% year over year on payload, on how many megabytes, but our signaling has grown 2,700%".

"It’s not just about getting more spectrum .. it’s about making it useable by the operators so they can build faster networks to deal with the onslaught of richer, bandwidth intensive applications being used over increasingly sophisticated mobile devices".

At the end of Q2 2012, Telus had 7.4M wireless subscribers; 39% of its ARPU came from data services during Q2, compared to 33% on the same quarter the year before (see chart).

See "IEEE: Traffic tsunami causing congestion in wireless nets, says Telus' Spadotto" - here.

NSN: MNOs/OTT Partnership Model - Quality for Revenue Share

A post by Carlijn Adema (picture), Nokia Siemens Networks’ Services team, discusses NSN's favorite subject - signaling / chatty applications / keep-alive messages problem and suggests to MNOs how to share revenues with OTT providers, in return for higher quality VoIP.

".. Recent results from a Nokia Siemens Networks Smart Labs study showed that mobile online poker can generate up to 4,000 extra signals per smartphone. When an app like this takes off, radio resources can be impacted, causing congestion and potential quality issues .. Nokia Siemens Networks can analyze these patterns and improve the network performance for smartphone users, helping to extend smartphone battery life and enhance the user experience. But there is even more…". See also "[NSN]: Chatty Android Phones should Move on to Network Controlled Fast Dormancy" - here.
  

Apple's FaceTime

"..Nokia Siemens Networks’ latest innovation in data service optimization is the optimization of VoIP services. Why? Voice still has a major stake in operator revenue, but VoIP solutions such as Skype and FaceTime are gaining popularity and putting operator voice revenue under pressure. With VoIP optimization, LTE operators can offer their own VoIP service. If operators consider other options, they can keep voice in GSM and 3G technologies but this will cut spectrum management flexibility whilst new ‘over the top’ (OTT) VoIP providers take advantage of their 4G data plans. In this case operators can decide to partner with an OTT provider, and offer quality through VoIP optimization, in exchange of a revenue share".

See also "[Infonetics]: Global OTT mVoIP Subscribers more than Doubled in 2011"- here and "European ISPs Suggest New Internet Business Models" - here.

See "Mobile operators: Are your mobile data services on a slippery slope?" - here.

Openet: US Tier1 MNO Signaling Optimization Win; Will Save $45M in 3 Years

Openet announced that "a tier one U.S. mobile operator has selected Openet Interaction Gateway (see "Openet Extends Policy Enforcement to the Device; Reduces Signaling Congestion" - here) for signaling optimization, resulting in projected savings of more than $45 million over three years".

AT&T is among Openet's US customers (here, here).

See "Tier One US Carrier Selects Openet for Signaling Management" - here.

Openet Extends Policy Enforcement to the Device; Reduces Signaling Congestion

Openet announced the ".. The immediate availability of Openet Interaction Gateway, which enables operators to extend Policy and Charging Control (PCC) system capabilities to the device for improved customer experience and optimization of network resources".

"Openet Interaction Gateway provides for device-level policy enforcement, enabling optimized device and network  interaction for a more granular control of services. The Interaction Gateway, in conjunction with an interaction agent  installed on the users handset, enables operators to monitor and manage application behaviour, set application priorities and manage network access requested by ‘chatty apps’". 
 
I asked Shane O'Flynn, Openet's Global VP Engineering who is providing the above mentioned agents: "we do for certain use cases and we use third parties for others where specific data is required or their agent is already in place"

"Interaction Gateway provides the advantage of enabling on-device enforcement of policies to reduce signaling congestion. By optimizing the signaling created by “chatty” apps that constantly query the network for updates, mobile device battery life can improve by up to 50 percent, and mobile infrastructure resources such as DHCP IP address allocation and AAA capacity can be preserved. Customer experience then improves through more robust mobile connectivity and better device performance".

See "Openet Introduces Interaction Gateway" - here.

SK Telecom Fights OTT Services and Signaling Load

SK Telecom continues its fight against OTT. After trying to protect their voice and messaging revenues by blocking access to competitive internet based services (see "Complaint against SK Telecom & KT for Blocking VoIP over 3G (Using DPI)" - here), the MNO is now launching its own services.

Mobile World l!ve reprots from GSMA MOBILE ASIA EXPO 2012 that "South Korea’s largest mobile operator SK Telecom (SKT) is fighting back against a number of Over-The-Top (OTT) players that have become hugely popular in the country in recent years, promising the imminent launch of an RCS-based service and further deployment of its Smart Push technology"

Dr. Choi Jin-Sung (pictured), SVP and Head of SKT’s Technology Strategy Office, told Mobile Asia Daily: “Korean mobile carriers are experiencing difficulties, including decreased SMS revenues and increased network investment cost to accommodate surging traffic, caused by the popularity of various free messaging services .. KaKao Talk, a mobile messaging service used by 37 million Koreans (46 million subscribers globally as of May 2012), is fast becoming a threat to mobile operators as it has recently launched a mVOIP service. And these OTT service providers are expected to evolve into a UC (Unified Communication) platform after securing subscribers based on their messaging services”

About the notorious signaling/keep-alive problem:

"At the same time, the company is applying Smart Push, its home-grown technology that enables integrated management of keep-alive signals, to prevent network overload .. Here, the opening up of the Smart Push technology is an exemplary case where all stakeholders including mobile carriers, OTT service providers and customers gain benefits .. SK Telecom has also entered into MOUs with Samsung Electronics and NSN to promote overseas exports of Smart Push-embedded network equipment”

See "SK Telecom fights OTT rivals" - here.

Optimization Deployments [138]:PCCW [HK] Reduces Signaling by 25% with Huawei

Nokia-Siemens Networks and Huawei continue to educate the market, push and sell solutions which address a significant pain for mobile operators - application level signaling traffic that floods the radio and backhaul links, creating congestion and reducing quality of experience for actual data.

Huawei announced that "PCCW mobile has adopted Huawei's pioneering smartphone signaling optimization solution to further enhance its network performance and improve the mobile broadband experience for its customers. The 3GPP-Release-8 (3GPP-R8)-based smartphone signaling optimization solution can significantly reduce 3GPP-R8 smartphone signaling traffic by over 25%".

Mr. Ying Weimin (pictured), President of GSM,UMTS and LTE wireless networks, Huawei, said: "It took several months for Huawei and PCCW mobile to finish the testing of our signaling solution on existing networks. The high performance of this solution shows that Huawei is capable of innovation to help customers succeed".

See "PCCW mobile Pioneers Huawei's 3GPP R8 Smartphone Optimization Solution to Reduce Signaling Traffic by over 25%" - here.

About M2F [Machine-to-Facebook] Mobile Traffic

Here is what Facebook says in its recent Registration Statement to the SEC (here):

"Our metrics are also affected by applications on certain mobile devices that automatically contact our servers for regular updates with no user action involved, and this activity can cause our system to count the user associated with such a device as an active user on the day such contact occurs. We estimate that less than 5% of our estimated worldwide DAUs [daily active users] as of December 31, 2011 and 2010 resulted from this type of automatic mobile activity, and that this type of activity had a substantially smaller effect on our estimate of worldwide MAUs [monthly active users] and mobile MAUs. The impact of this automatic activity on our metrics varies by geography because mobile usage varies in different regions of the world".

This may not be significant in terms of traffic, but joins the general phenomena of "signaling" /"keep-alive" traffic, in different communication layers. However, it does impose load on mobile networks, for example in the number of concurrent connections/PDP contexts etc - with questionable value to the user.

Another example at the application layer: VoIP (see "NTT Explains Outage and its $1.5B Corrective Action" - here)

NTT Explains Outage and its $1.5B Corrective Action

 
Last month NTT DoCoMo experienced network outage due to smartphone signaling storm originated by an Android VoIP appication (see "VoIP Signaling Crashed NTT DoCoMo; Asks Google to help" - here). 

So probably help from Google was not enough, and NTT decided to "optimize resource allocation of packet switches, to expand the number of signals that can be processed", by mid August 2012, for ¥2B, and to "expand the capacity of packet switches and increase installation of packet switching system capable of accommodating 50M smartphones"  for another ¥120B, to be completed by 2014. Both totaled to $1.5B, and NTT also provided an explanation of the problem itself.

NTT decided to solve the problem by expanding network capacity, however vendors claim for a better solution (although I am sure someone is happy with the "brute force" solution) - see "[NSN]: Chatty Android Phones should Move on to Network Controlled Fast Dormancy" - here. 

See "Response to Series on Network Malfunctions" - here.

[NSN]: Chatty Android Phones should Move on to Network Controlled Fast Dormancy

Nokia Siemens Networks continues to provide valuable information on traffic loads caused by signaling (see a related recent post - "VoIP Signaling Crashed NTT DoCoMo; Asks Google to help" - here).

This time, an article by Gerald Reddig (pictured) from Mobile Broadband Marketing, looks at ".. existing 3G smartphones that have been tested in our Nokia Siemens Networks Smart Labs using OTT Voice over IP applications Nimbuzz, GoogleTalk and Skype, the issue seems to be more about different smartphone behavior. This includes the signaling load during a Skype voice call on Samsung´s Android phones compared to Nokia phones and iPhones".

"The Android phones generated 600% more signaling traffic during VoIP calls .. These impressive results have recently convinced a number of device manufacturers using an Android operating system (OS) or a Blackberry OS to step back from proprietary device driven Fast Dormancy and move on to standardized Network Controlled Fast Dormancy .. As always-on OTT applications proliferate, developers and handset manufacturers will increasingly have to look at new ways to reduce signaling load".

Source: Nokia Siemens Networks

VoIP Signaling Crashed NTT DoCoMo; Asks Google to help

    

Caroline Gabriel reports to :rethink wireless that "NTT DoCoMo has added its loud voice to the wave of operator criticism of how smartphones overload carrier networks with their 'signalling storm'. The Japanese cellco recently suffered a major outage in its network, which it is attributing specifically to an Android VoIP application, and more generally to the way Android constantly polls the network, even when users are inactive, generating huge levels of signalling traffic"

While DoCoMo goes to the source of the problem (at least one of them), there are also some ways to address the problem from the network side. Nokia Siemens Networks has been very active in this space. See "NSN: "VoIP Applications Create Huge Signaling Load" - here and "NSN: Android and Blackberry Phones Overload Networks with Signaling" - here. 

"DoCoMo, according to the Nikkei news agency, has demanded that Google rein in the signalling and data loads imposed by Android, particularly the habit of handsets transmitting control signals to the network, and pinging the servers, automatically rather than as-needed. These behaviors support constantly updated apps such as social networks, as well as the rising tide of cloud-based software, in which data and apps are held on the server and constantly sync with the device".

See "DoCoMo demands Google's help with signalling storm" - here.

NSN: "VoIP Applications Create Huge Signaling Load"

 
Nokia Siemens Networks educates the market and pushes its solution for the mobile signaling problem for long time now, as many operators say it is a major problem both in terms of performance (traffic and long-living sessions) and well as battery draining issue for subscribers. This becomes even more relevant to NSN now, as mobile broadband becomes its main business focus (here).

A new post to the vendors' blog by Gerald Reddig (pictured), Global Marketing Manager, shows how "A European 3G network is benefiting from a 30% signaling traffic reduction, three months after Network Controlled Fast Dormancy (NCFD) was implemented by Nokia Siemens Networks ..  Nokia and Apple decided early on to support NCFD on their smartphones [here], with Blackberry [here] and Android OS device [here] manufacturers like Samsung, LG and Sony Ericsson following suit"

",, Standardized in 3GPP Release 8 [see below], Nokia Siemens Networks’ Network Controlled Fast Dormancy or NCFD aids by acting as a more intermediate state: one with less signaling and faster reconnections"

"Nokia Siemens Networks’ Smart Labs [here] tested popular over-the-top (OTT) VoIP applications running on different smartphone operation systems. One of the results was the huge signaling load caused by VoIP applications in non-NCFD enabled smartphones – 600% more signaling traffic"

See "Faster connections, longer-lasting batteries: A fifth of all smartphones now benefit from Network Controlled Fast Dormancy" - here.

See also - "T-Mobile Netherlands Reduces Data Signaling by 30% with NSN Cell_PCH" - here and "SFR Deploys NSN Cell_PCH Optimization Technology" (here)

GSMA: Fast Dormancy Best Practices

NSN: Will 3G Signaling Problems Repeat in LTE? Maybe

 

I had a number of posts regarding the signaling problems and solutions in 3G networks (here) - an issue that Nokia Siemens Networks (here, here, here) and Huawei (here) are actively promoting.

A post by Vijay Sankaran (pictured), NSN, discusses whether we will face this problems in LTE as well:

"Will signaling require as much resources and attention in LTE as it has in 3G? The short answer is:  Definitely not!  . . . .maybe. The “Definitely not!” part of the answer comes from the point that LTE is structured quite differently from HSPA, and so LTE signaling is handled differently ..  However (and this is the “maybe”), it’s important for us to emphasize that we don’t have the end to end LTE smartphone environment in place yet, since by and large, the end device is still missing" : The good news is that through the Nokia Siemens Networks Smart Labs, we’re working with handset manufacturers to ensure that they understand the issues around the generation of excessive signaling now"

See "Will Signaling be a Problem in LTE?" - here

NSN Helps Reducing Irrelevant Blackberry Signaling

 

Nice story by Vijay Sankaran (picture) from NSN about the infamous "keep alive"/signaling problem and how NSN's Service Assurance solution and configuring the SGSN while using the Subscriber Data Management solution solved it.

Some background - "NSN: Android and Blackberry Phones Overload Networks with Signaling" - here.

"The operator was puzzled when they started seeing a huge increase in signalling traffic, at a much faster growth rate than had been observed previously.  The growth was so rapid that it very quickly threatened network stability, so something had to be done fast ..  It turns out that, unlike other handsets, BlackBerries are pre-programmed to connect to the network and to look for email updates automatically, whether they have an active data plan provisioned for them or not.  And this makes tons of sense for those people who buy BlackBerries to stay up to date with their mobile email.  But this operator’s “BlackBerry with no data plan” offering had attracted a lot of SMS users who wanted a keyboard but weren’t planning on connecting to the internet."

"How did the operator fix it?  Two steps:  first of all, by creating an attractive, targeted data plan for these recent BlackBerry acquirers and marketing it to these customers directly (we used our Campaign Management solution to do this).  Second, by identifying which users had BlackBerry models and configuring the SGSN (Signalling Gateway) to ignore signalling coming from those handsets if they had no associated data plan (this was done with our Subscriber Data Management solution)."

See post - "Always sell BlackBerry with a data plan!" - here.