Internet traffic as cost driver for telcos - not

One of the longer-running debates in the Internet infrastructure area is the cost impact of Internet services and content delivery by 3rd party providers on local ISPs and telco's IP access network and peering points. Or, taking the opposite view, how 3rd party services and content delivery from the likes of Google/YouTube, Netflix and for instance via the Akamai CDN, has been the key driver helping local ISPs and telcos in actually selling and getting traction for their Internet access services and bundles.  No attractive Internet services and content over the Internet, no local uptake in telco broadband offerings and Internet access business to the residential and business market...

I won't reference the main cases and "conflicts" going on in this area, as I'm sure readers of this blog are mostly familiar with the YouTube and Netflix peering and transit discussions with local or regional ISPs, but rather look into the argument that network build and provisioning of network capacity to 3rd party Internet services and content delivery (that is, 3rd party to the local ISP and telco) are a main or large cost driver to the local ISP or telco.

Firstly, a very rough guide to Internet services or content delivery, using the first, middle and last mile analogy:

1. First mile Internet services and content delivery: All things involved with service and content design, build and preparation for delivery, storing and hosting the service or content in one or several data centers, one or several clouds, for service and content origin.  Internet access capacity from the hoster on to the Internet for service and content distribution.
• Many ISPs and telcos are in the data center or hosting business, so this a source of income for many ISPs and telcos.
  
2. Middle mile service and content delivery: All things to do Internet/ISP traffic exchange (free peering, paid peering, paid transit) and service/content delivery via one or several Content Distribution Networks (CDNs) or via cloud providers. Handover to local access network for end-users, operated by local ISP or telco.
• As above, many ISPs and telcos are in the Internet peering (achieving Internet traffic aggregation for improved transit position towards peers, not necessarily direct money), transit, CDN and/or cloud business themselves mostly through wholesale set-ups, so this is also a source of income for many ISPs and telcos.
  
3. Last-mile service or content delivery: All things to do with delivery of services and content to the local end-user, by fixed or mobile broadband.  Will in many cases be done via a mix of locally deployed 3rd party CDNs or ISP caches, a mix of layer 3-5 load balancing by 3rd party or ISP.  And for content provisioning, adaptive streaming taking end-user bandwidth and client processing capability into the equation (negotiated with content origin or CDN edge caches in question).
• Main source of income for most ISPs and telcos, based on selling local access bundles, 3rd party service add-ons and ISP bundled services like TV or IP in various forms, music services, sponsored tablets and smartphones tied to broadband subscription of some sort and voice over IP services. 

Looking at this, it's possible to make the argument that ISPs are uniquely positioned to cash in on and control the distribution of 3rd party services and content deliveries )(overlooking that Internet majors seldom hosts with local heros that much), but still there is dissatisfaction with how 3rd party service providers and content services like Netflix and YouTube wrecks ISP and telco backbones and economies. Why is that?

Looking at ISP cost categories and drivers, they can be divided as follows (not necessarily in order of cost impact):

1. FTEs and man-hours: Skilled personnel needed to design, deploy and operate quite complex network structures in the backbone and local access areas, protect from Internet hacking and DDoS attacks.
2. Network equipment: Routers, switches, firewalls, control planes, backbone fiber, local access network, POP and (over-)capacities at major national and international peering and transit locations, DCs and co-lo space for network equipment.
   
3. IT IS and back-office IT systems for service authentication, service provisioning, billing and help-desk.  A major cost driver for most telcos and ISPs, as service complexity, number of integration points, legacy IT IS systems and integration of virtual network functions seems to multiply out of control weekly.
   
4. Internet transit and peering capacity, off-net capacity: ISPS and telcos needs to buy Internet CDN, peering and/or transit network capacity with or from 3rd parties. This is the cost element that is most often brought forward by ISPs and telcos in the "OTT is eating my network" debates and arguments.

Looking at these costs elements and drivers, one can argue that

1. FTEs and man-hours per ce are under control, and going down due to outsourcing of many network operations tasks. Key challenge seems to be number of FTEs needed for a certain operational capability, not the salary levels per FTE in itself.
   
2. Network equipment: Overall trend is for this kind of equipment following Moore's law, giving ISPs and telcos ever more processing power or networking capacity. Same downward price points for DCs and co-lo.
   
3. Internet peering/transit and CDN price points are involved in a race to the bottom giving Mbps or that GB/month pricing that would have been unthinkable some few years ago.  Although a bit old, have a look at "Internet Transit Prices - Historical and Projected" and for CDN pricing, "CDN Pricing Stable: Survey Data Shows Pricing Down 15% This year"

This leaves IT IS and back-office IT systems as the fall-guy, and here the legacy plus complexity spiral has most companies, not only ISPs and telcos, but banks, insurance, governments and institutions trapped.  IT IS remains a key cost driver for most companies, whereas Internet giants like Amazon, Google, Netflix and Facebook has achieved rapid service development, service provisioning and roll-out and consistent service performance (equals commercial success) precisely because they have managed to get their IT IS systems under control or rather, into a flexible and adaptable software development process environment being at the forefront of cloud developments. And IT IS then becomes a competitive and cool tool rather than some boring way in the back back-office legacy and career ending thing.

An additional argument that can be made is also the overall cost savings ISPs and telcos have been able to achieve using Internet and tcp/ip technologies (IP on everything, everything on IP...), and not things like pre-Internet ISDN and ATM, OSI-stacks, for integrated services networks and one network for multiple services that we all take for granted today. Imagine a competitive IT IS stack for OSI on ATM versus one control plane for IP-based networking.

In summary: ISPs and telcos stand to gain the most from adopting SW development processes, environments and cloud technologies on par with Internet giants to simplify their IT IS stack, speed up service delivery and competitiveness.  Not being competitive in this area is a far greater threat than Internet 3rd parties consuming too much bandwidth without paying for it.

Erik Jensen, 27.1.2014

The net neutrality that never was: Bringing net neutrality up to speed

Lot of comments and writings has been made after the US Court of Appeals for the District of Columbia Circuit last week halted the FCC's "ban on traffic blocking and discrimination by Internet service providers because the FCC had not designated ISPs as common carriers" (*). Most of the writing has fallen into the camp that this more or less means that non-discrimination of Internet traffic and net neutrality for Internet traffic and parties is dead and that this signals the end of basic Internet traffic delivery freedoms. In the US. And seemingly most other places as well.

In the US, were most areas and cities are only served by one dominant service provider or telco, the situation in many ways are more dire than most other well-functioning markets, where there are multiple Internet access providers and access technologies to choose from, and one dominant access provider can't run amok in how basic access services are provisioned.  And, let's say, play favoritism with one's own services or content offerings.

That said, I think the basic Internet neutrality approach and arguments has been flawed from day one, or at least half of what net neutrality implicates.  To me, net neutrality could and does entail two things

1. Neutrality as to how network protocols, services, applications and devices are treated, i.e the same and without prejudice or helping hands of any sort
   
2. Neutrality as to how communicating parties are treated, i.e. basically how sender and receivers or peers in a networking session are treated, without prejudice and equally.  For instance
• Broadband Internet access customers treated the same for the same service or session
• Content providers, Internet service providers treated the same towards broadband subscribers and in IP backbone management

Number 2 is taken for granted by most parties I believe, whereas number 1 I think was and is a failed approach as most network protocols, Internet services and applications as well as end-user devices never was or is designed to be "neutral" or equal to one another, nor to they behave particularly "net neutral".

A couple of examples as to why "network protocols, Internet services and applications as well as end-user devices" are closer to the "all you can eat" camp or passive resources, than nicely behaved, neutral net citizens:

1. Some layer 5 tcp/ip protocols are better designed or designed for better performance than others (depending on how old they are, what they are designed or forced to support or utilize the underlying tcp/ip stack).  Some tcp/ip protocols are also better at utilizing or able to use sliding window mechanisms, giving them larger transmissions windows and better abilities for avoiding IP traffic congestion.  And there are of course performance and behaviour differences in how he basic tcp/ip stack is designed and implemented in differents operating systems and network elements.
   
2. Use of http for adaptive media streaming means streaming clients will quite aggressively seek to reach the highest encoded video rate achievable for their broadband connection and media device playback capability, meaning other network protocols and service deliveries as well as non-adaptive devices will suffer and be put to the back on the broadband connection.
   
3. There are few or none IP packets that traverses the Internet between a sender and a receiver or between two peers today that hasn't been modified in some way by one or more load-balancers, NATs, layer 3-5 traffic accelerators, a cache server or CDN service or had their DNS/IP header re-written or payload compressed in some way or modified depending on OS, browser, location or time of day, meaning that full-time network modification or alterations to Internet traffic is already a fact of life.  And was also some 6-7 years ago. MOst of these are very useful and necessary developments for IP and Internet traffic management, scaling and optimization as the basic tcp/ip protocol stack is old.  Http 1.1 is also getting geriatric. Meaning layer 3-5 protocol optimization, caching and application level load balancing are useful and necessary additions to basic, "neutral" IP networking.
   
4. Most Internet traffic and content today are served by a couple of Internet giants.  As measured in the ATLAS Internet Observatory 2009 Annual Report, "In 2009, 150 ASNs contribute 50% of all Internet traffic".  Would anyone be surprised if some 25-50 ASNs are behind closer to 75% of Internet traffic in 2014?  These Internet giants are using their own fiber backbones and cache/CDN/cloud infrastructures for service and content delivery towards end users, not the basic Internet itself, and Internet bypass for first-mile and middle-mile service delivery has been the default approach and get things done in large-scale settings for years.

These are some of the ways basic Internet network protocols and service delivery never was or isn't "neutral" from the outset, and with cloud-based networking and service delivery becoming the norm towards broadband users and customer in both the business and residential market, this "non-Internet" approach will only accelerate in the coming years.  

As noted above, many of the advance in layer 3-5 traffic optimization, acceleration and load balancing greatly helps in the delivery of services and content towards end users.  And I would like to see the user who takes his YouTube videos or mobile app usage without any traffic assistance at all and opts for a "neutral" non-QoS delivery (it's not even best-effort) over a optimized and accelerated service delivery.  

For net neutrality I think the main focus for policy development should center around 

• Treating everyone the same, including giving everyone the option of paying for optimized transport and Internet traffic management on Internet first-mile, middle-mile and/or last-mile sections
• Open and transparent information as to how first-mile, middle-mile and last-mile service providers (including CDN- and cloud operators) utilize Internet traffic management and optimization technologies for their networks, CDNs or clouds and what's available for 3rd parties to utilize for their service delivery through open APIs, SDN interfaces or manually.  In short, what's their internal and 3rd party QoS regime and policies.
• Don't limit IP network, Internet and IP traffic management developments by enforcing a net neutral straitjacket on Internet networking and service delivery that hasn't kept up with reality for the last 10 years or so.

In short: The net was never neutral, but users on the Internet needs to be treated neutrally.

Erik Jensen, 20.1.2014

Why doesn't mobile Internet access translate to mobile revenue and increased ARPU?

Mobile Internet access and traffic - there's no shortage of it.  In fact, in the latest Sandvine Global Internet Phenomena Report for 2H, 2013, mobile traffic is shown to grow rapidly for every market being monitored.  For instance:

• North America, since 1H report: Mean monthly usage has made a 13.5% jump, increasing from 390.1 MB to 443.5 MB
• Europe: Mean monthly usage has increased 15% from 311 MB to 358.4 MB per month
• Asia: Increasing from 700.4 MB to over 1.1 GB per month

Overall, mobile traffic (i.e. from mobile devices) only makes up some 20% of Internet traffic (see KPCB 2013 Internet Trends, slide 32), but growing at 1,5x per year or more, mobile broadband traffic is on a pretty good trajectory.

Similarly, the Ericsson Mobility report for November 2013 states that "the increase of monthly mobile data traffic in Q3 2013 exceeded total monthly mobile data traffic in Q4 2009" and that there was a "80% growth in data traffic between Q3 2012 and Q3 2013".  And looking forward, "mobile data traffic is expected to grow at a CAGR of around 45 percent (2013-2019) leading to a 10x growth in mobile data traffic between 2013 and 2019".

So, volume-wise, mobile traffic numbers look good if you are in the mobile broadband area or a mobile access provider.  Does the traffic increases translate to money for mobile access operators?

While mobile traffic is increasing significantly, ARPU for mobile operators, for both mixed voice and mobile broadband providers as well as voice only providers,  are expected to fall in the next years (GSMA and OVUM numbers a bit old):

• GSMA: "European mobile ARPU falls 20%", 1, 2
• Global Mobile Outlook, OVUM (2011): Monthly ARPU remains in steady decline across all the regions, and we expect it to fall at a CAGR of 4% from 2011 to 2016.
• The Mobile Economy 2013, AT Kearney: Despite the growth in usage of voice and SMS and increasing numbers of data subscriptions, ARPU rates have declined across every region globally. The overall global ARPU rate has fallen by 7.6% p.a. from US$19 to US$14 per month, with the highest reductions in 2010-2012 seen in Africa (-10% p.a.) and Europe (-7% p.a.)

So, revenue per user going down for mobile operators, but average traffic volumes per subscriber going up rapidly.  Why?  There's a number of reasons among them:

1. Despite Moore's Law and networking units getting ever more capacity at ever lower prices, functionality, complexity, service management levels and operations transactions per networking unit is going up.  There is seldom less complexity introduced with new mobile G generations and always-on mobile units, costs for building and maintaining mobile networks are also increasing.
2. Race to the bottom.lack of service differentiation: Telco and mobile operations has squarely entered the mass-production or utility service era, and there is little noticeable service quality or service range differentiation once basic mobile coverage has been ensured. Operators are competing on level of mobile handset subsidies and traffic volume/capping bundles.
3. Mobile access subscription and service does not translate to any significant degree of share of "mobile wallet and spend" - customers get their mobile service needs by Internet and 3rd party providers.
4. Lack of investment or market foothold in first-mile services (i.e. hosting, extended comms and messaging) and middle-mile infrastructure and services (i.e. CDN, on-net traffic aggregation) for mobile operators and telco's .

Point 4 can be illustrated by how things are looking in the mobile ad space and mobile app & service subscription space - much better than mobile voice and broadband ARPU developments! Some indicators:

1. Business insider,THE FUTURE OF DIGITAL, 2013: Mobile is the only media time that's growing (slide 24), Mobile is now approx 20% of e-commerce traffic (slide 37)
2. Mobile ads now close to half of Facebook ad ARPU - slide 37 in the KPCB report referenced above
3. Twitter: " More than 70 percent of advertising revenue came from mobile devices in the third quarter, compared with 65 percent in the second quarter." (2013 numbers, of total Twitter revenue of $168.6 million in the last period).
4. Gartner is predicting that worldwide revenue from app stores will increase this year (2013) by 62%, bringing the total industry revenue to $25 billion dollars.
5. Business Insider, "The Results So Far From Holiday Shopping Point To Huge Gains For Mobile Commerce This Year":  ...mobile commerce grew more than twice as quickly (as e-commerce overall), at 63%, and accounted for nearly $940 million in sales on those three days. ...one in four e-commerce dollars spent on Black Friday and Thanksgiving were on purchases made through mobile devices. ...Tablets have emerged as a principal engine for mobile commerce growth.

So, to answer the question raised in the title: It turns out there is no direct relation, so far?, between providing mobile Internet access and capturing share of mobile traffic and services revenue, or share of mobile customers spend on e-commerce, Internet services and use of traffic-intensive deliveries.  Because with whom customers choose to have their mobile access isn't the same parties that they choose to use for e-commerce, Internet services like messaging, social media sharing, file sync, photo storage and get delivery of movies and entertainment. 

Erik Jensen, 15.1.2014