Cross-functional teams and assemby lines

Assembly lines. Used since the dawn of the industrial age to produce one thing repeatedly over and over with as little cost as possible and as quickly as possibly. The thing could be simple object, let's say an hammer or an object made out of several components and parts, moving or bolted solidly together, fr instance a car.

One way to speed up the assembly line and rate of output rate was to divide the assembly of the parts of an object into a series of separate, repeatable work steps, performed by different persons or teams along the assembly line. In this way, a set of tools and the parts required for the assembly could be stacked along the assembly line.  This in turn lead to the specialization of work tasks along the assembly line, and it became possible to measure the output per work step (number of parts assembled), the failure rate and, not the least, the time different workers or teams spent on their section of the assembly line.  To improve quality levels and outputs, teams were trained for their special assembly task, and baseline metrics set for number of assemblies to be made per hour or workday.  In this way worker and team specialization were born, and we got organisational compartmentalization ("to separate into isolated compartments or categories"), were the workings of one functionally aligned team shouldn't interfere with others, but just interact along clearly defined borders, interfaces or hand-over points.

Work specialization, grouping of repeatable tasks and organisational compartmentalization has been going on ever since, providing a especially good fit in hierarchical organisations, i.e. most businesses, where span of control, costs of goods sold, margins and performance tracking of each employee are part of everyday life for quarterly results and internal business reviews.  Production units can be measured, monitored and reported upon with a set of basic production KPIs and cost baselines consistently over time (down to the minutes if one wants). making it easy to identify good and bad performance over time per unit/compartment.

So far so good for most, or certainly for accountants or the finance dept. For organisations doing the same thing over and over this proved a unbeatable model.  For organisations and businesses that provided a range of customer specific goods or services, or relied on a heavy dose of product development to constantly field the market with a series of new products or offering to stay competitive, not so much.  Compartmentalization doesn't play well goods or services that needs a degree of tailoring for a set of customers, that are dependent upon flexible financing and wide range of competencies to understand market and customer, innovation opportunities and open communication across organisational units.

Enter DevOps cross-functional teams, employing agile work methods and team workings. For our discussion of "structured, compartmentalized work units for assembly lines", it's interesting to compare the approach and mindset in the Agile Manifesto with compartmentalization:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

Anyone doing a production line for standardized assembly would aim for right-hand side and feel quite foreign to left-hand side.  Or it wouldn't ever cross their mind. 

Anyone doing service development, software development, supporting customers with customer-specific services (sometimes legacy services even) or portfolios, or anything that can been seen as competency-intensive requiring a large degree of organisational flexibility, would start left-hand side. As "individuals and interactions", "customer collaboration" are a huge part of reaching "organisation flexibility", attracting and retaining top talent and evolving cross-functionalities with customers to stay competitive.

The key feature of DevOps and agile mindset, culture and work methods are the cross-functional composition of teams, consisting of members with different competencies and professional training, project experiences and the aggregated experience they bring to the team in their role and task at hand. In fact, putting together teams with team members having widely different and alternative, for the task at hand, competencies, work experiences and background are seen as a key dynamic for the cross-functional team. Different opinions, ways of working and skill sets will produce more dynamic and interesting results than monolithic, coherent teams.

Part and parcel of the cross-functional team setup are entrusting the teams with a large degree of autonomy on how a task is done, what the actual output can be and even which tasks or business challenges they choose to work on.  A cross-functional team might be tasked with a specific product development area or business function, for instance customer facing services, but ultimately the team decides which area to attack how, with which tools and what the key deliverables will be when per their product backlog.

Cross-functional teams doesn't easily lead themselves to be monitored, base lined and reported as monolithic teams, making them somewhat an outsider approach for many traditionally rigged businesses that are brought up with waterfall project ways, tight line of command & control and budget control as the key management vehicle. 

As "digitalisation" becomes an integral part of most business strategies, it will be hard for most businesses not to employ cross-functional teams and competencies to thoroughly perform and understand multi-channel customer journey mapping, analysis and mapping to service development and product management of big data collections and impact of "digitalisation" on organisation and management of the business.


Erik Jensen

September 6th, 2017

- Ke Jie is playing perfectly at the moment: Automated reasoning and risk/performance assessment

And the full statement was "Incredible. According to #AlphaGo evaluations Ke Jie is playing perfectly at the moment." By Demis Hassabis, @demishassabis, founder & CEO of DeepMind.

And per Wikipedia, Ke Jie is a "Chinese professional Go player of 9 dan rank. As of 2 June 2017, he is currently ranked number one in the world .... He continuously held the world No. 1 ranking for two years from late 2014 to late 2016.  In early January 2017, Ke Jie unofficially played "Master", an updated version of AlphaGo online, losing all three games."

So we've got AlphaGo, i.e DeepMind playing or optimized for Go, also passing judgement on opponents and human performance at any given time, based on awareness of others. At least their score against some benchmark or predefined structure.

This could imply some sort of self awareness, at least awareness of others and how they are performing, but most likely AlphaGo was rating opponent based on statistical assessment and rating of Go moves available to Ke Jie.

Still the self-awareness angle or probability remains close and interesting, and it's easy to foresee a not to distant future when a cognitive bot or automated reasoning entity of sorts will rate any ones daily/hourly work or sports performance, your homework or semester paper, driving ability or health against a personal, social or company KPI. Or inside a Gauss-scale or a Myers–Briggs Type Indicator scale.

AI entities with some kind of self awareness (or on the hunt for self awareness, memories or self) are very much the key plot of many sci-fi books and movies, including the Aliens series, Bladerunner and 2001: A Space Odyssey.

Moder in Aliens: Covenant certainly could use a healthy dose of assertive self and automated reasoning to avoid human disasters and bio-hazards high and low.  Moder "defects" include, and the time is year 2100 no less in Covenant, little or no predictive reasoning, no risk assessment capability, no android or IoT tracking, no bio-hazard analysis entering new planets and allowing people or bio samples on-board without any bio scanning. And why not deploy a drone or 2 to check out the environment?

Strangely, the androids in the Aliens series are much more astute and self-aware, human-like, than the rather standard sci-fi fare cognitive capabilities that Moder steering the spaceships brings along, but I suspect that is plot malfunction or not working this side of the plot through thoroughly.

Replicants in Bladerunner were looking for inception date determining their life span and, in the end, what their pictures and memories from childhood were - implanted or real?  They had self-awareness and insight in spades, but if memories were RAM implants only, the stories they told about themselves were fake and their self-awareness were imaginations, RAM patterns.

HAL 9000 in 2001: A Space Odyssey had too much self-awareness and ability to judge others and himself, ultimately leading to his own shut-down when conflicting knowledge about crew plans (shutting him down), reasoning and mission orders led to premature jettisoning of humans from spacecraft. The crew in the 2001 movie might have "been playing perfectly at the moment" in the execution of their mission, but HAL had other benchmarks to judge their performance against, i.e. checking out that black monolith on Saturn if I remember correctly.

That's movies.  If one goes back to René Descartes, the self awareness baseline and starting point will be his "Cogito ergo sum" proposition, typically translated into English as "I think, therefore I am".  Self awareness and reasoning about oneself results from ability to formulate ideas, having cognitive processes, testing and observing those processes over and over again to reach "probabilities" for the outcome of a initiated process.

Current "mechanic" or pre-structured machine learning against a bound data set, big or small, not there yet for self-awareness, ad lib reasoning and  assessments, but future automated reasoning surely will expand on these capabilities.

When machine learning systems goes to deep learning and then reaches stage of autonomous and automated reasoning, these systems will probably one day have their cogito ergo sum moment also, but with structures and thought patterns strangely different (or much more logical) than humans.
Rating of and education for human performance will be based on non-human parameters.
And that brings us back to Ke Jie again and his great summary following his +20 Go wins since the AlphaGo match, please see here.


Erik Jensen
12.7.2017

10 things about Internet of Things

10 things about Internet of Things

1. All the IoT devices will have their own friggin lasers and LEDs, each blinking in an annoying POST and live patterns of their own.
   
2. Each IoT device and vendor will have their own end-user agreement starting at 10 pages or so, "enhanced" with 5 additional pages quarterly
   
3. Each IoT device will make different noises and plings (alerts, I'm alive, notices...) all day and all night long.
   
4. IoT device firmware will be updated Samsung-style - every now and then first 12 months, then never.
   
5. Just like the NSA and Facebook, there's not a single set of data that is deemed out of bounds and that a IoT vendor will refrain from logging and storing somewhere for a "improved user experience".
   
6. The goal of IoT is to collect user data and sell in situ ads
   
7. Each IoT device will use their own IP port and range of external access options will come as a surprise.
   
8. IoT security is an add-on, not built in.
   
9. 3rd party access to IoT devices and IoT logs will give you cheaper home/car/family/health insurance and become mandatory after a mandatory voluntary period.
   
10. The vendor with the best IoT back end and ecosystem for 3rd parties will win.

Going cloud, going offshore - it's all about IT automation

Two major trends impacting enterprise IT worldwide is

• The move towards cloud-based IT service delivery (from dedicated or virtualized server deployments inside customers data centers or with 3rd party DC operators), and
• Global sourcing of IT operations and service delivery, or off-shoring when handing over IT operations and delivery to "offshore" IT suppliers

Both of them are concerned with or aims to, I would argue, reduce IT cost through increased IT automation.

The journey from legacy, classic server environments to cloud-based IT service delivery models can be depicted as in fig 1 below.

Here, various IT systems with a enterprise customer or let's say with a IT service provider, are in various stages of being run on classic, legacy IT environments on virtualized server platforms or maybe has made the leap to a VM IaaS/PaaS setup with a cloud service provider.  

For legacy IT environments, typically one IT system are running on a dedicated hardware and database platform, and IT budgeting is usually done based on CAPEX upfront, 3-4 years write-off period.  Legacy IT environments would be based on IT systems popular 10-20 years ago, i.e. SUN servers running Solaris, HP servers running HP-UX, IBM servers running AIX etc, even pre MS WinServer pre 2012 server on COTS Intel servers.

Some of these IT systems or business applications running on them would be able to make the leap onto virtualized IT-platforms, either as part of a major server consolidation projects that most enterprise IT departments would be running since the dot.com era. Leading candidates or candidate of course being VMWare, with freeware hypervisors like KVM and Zen, additional hypervisors form Citrix, Parallels and others being utilized also.

When a critical mass of virtualized servers were reached, the creation and utilization of server instances could be viewed as OPEX and on a per month, week basis, the cost of each new VM being incremental.  Also when a critical mass of CPU cores on a number of physical servers were reached, it was each to provision VMs from a pool of available CPU cores, over-commit on VM cores that were scheduled to being put into production or assign cores to VMs for dedicated VM resources and instances.  At a higher price than pool-VMs. 

With virtualized servers, server or VM provisioning and configuration could be dramatically automated and lead times for VMs were/are dramatically lower than installing dedicated, physical servers - seconds and minutes versus days and weeks.  VM re-configuration could also be done on the fly with "instant" re-sizing of VM CPU cores, memory and disk-space changes and allocation.  A significant higher degree of automation and IT-production levels were achieved with virtualized servers, leading to lower IT cost overall (per VM, per IT employee, per production unit etc).

Some IT workloads and IT systems has made the transition onto private or public cloud infrastructures, leading to a even higher degree of IT automation than traditionally available from both virtualized or legacy IT environments,  Between highly virtualized and automated IT environments and cloud based IT-delivery there aren't really a clear cut switch-over or demarcation line, but I guess cloud based IT delivery are seen as having a higher degree of auto-scaling and capacity on demand than a single location VM-environment.  Plus a higher degree of self-serve support and options for IT management than a on-prem solution.  IT departments did server virtualization for themselves and to meet corporate cost targets, while cloud IT delivery are available to a wider audience with an associated price plan, service catalog and SLA accessible in a way not always seen with corporate IT departments.

For many end-users, business applications delivered as a SaaS-solution, represents state of the art in automated IT delivery, "just" insert the data and press play. While cloud IaaS or PaaS-delivery would be state of IT-automation for IT-departments and developers.

In many ways, outsourcing and offshoring of IT operations and service delivery, can be seen as an IT automation drive also.

If we apply a onshore (on-prem) and offshore dimension to illustration above, we have a lineup as depicted in figure 2.

Corporate IT systems are, in addition to various state of "physical to cloud"server platforms, in different states of being managed and operated onshore locally (on-prem with customer, with 3rd party local IT-provider) or with a offshore, seen from the customers point of view, IT provider performing day to day operations, maintenance and incident management.

These day to day operations and maintenance work are being performed inside or based on well-defined work packages and by personnel that has specific module certifications on various Microsoft, Oracle, SAP, HP, RedHat, EMC/VMWare etc IT platforms and systems.  In turn this means that IT management are removed from specific personal skill-sets or knowledge, and one set of work tasks lets say on a Oracle DB can be performed interchangeably by different Oracle-trained personnel, and one reaches a new level of IT automation where the personal/personnel factor is taken out of the IT operations equation.  Work tasks gets increasingly specific and well specified, customers avoid customer specific adaptations and developments as far as possible, i.e IT work and delivery gets boxed in and turned into work modules specified down to the minute.

Put another way, part of the cost benefit of offshore IT delivery are down to the modularization of IT work tasks and IT operations that offshore providers have achieved compared to in-house IT,

Thus the transition B in figure 2 is part of an overall mega-trend that uses IT automation to reach lower IT production costs, and it will be interesting to see how the IT service delivery business unfolds between offshore IT-providers and cloud-based IT delivery.  Or more likely, how offshore IT-providers use cloud-based delivery options (their own private cloud services, mix of public clouds) to reach new IT automation levels and increased market share.


Erik Jensen, 19.11.2014

Not in Kansas anymore: How the new Internet and cloud-based networking impacts network neutrality

In an article in the Norwegian newspaper Dagbladet.no June 11th 2014, Harald Krohg of Telenor calls for an discussion on net neutrality that starts with how the world of Internet actually is and not how it used to be some 20 years ago (largely an academic arena - it wasn't, but that's another story).

What might this "new" Internet actually be and why is it any different from earlier versions? And what has this to do with the network neutrality debate that the article in question discussed? Lets look at some major changes in the fabric of the Internet over the last 7-8 years or so, and some of the new developments that would potentially change the piping of the Internet and how services over the Internet are delivered and consumed.  And not the least, how these new developments impacts "network neutrality".

I guess most people with some interest in Internet traffic exchange and flows, has learned that the Internet is a series of interconnected networks operated by different network service providers, operated autonomously and independant of each other, exchanging traffic with each others as peers (i.e. more or less equals) and covering costs this might incur themselves. Or buying full routing and access to the global Internet through Internet transit.

This could be called the classic Internet, and was more or less how the whole global Internet came about in the 80's and 90's; thousands of independant network service providers worldwide linking up with each other, exchanging traffic as peers and buying transit to networks or parties they didn't have immediate access to. Or didn't want to manage with a direct peering relationship.

For a number of years the Internet developed as a many to many interconnected web of networks and within structured tiers: There were tier 1 large internationals, tier 2 regional, medium network operators typically and tier 3 small, local network operators organized in a structured network hierarchy, but close to all parties were able to exchange traffic with each other through mutual peering or paid transit (usually to tier 1's or some tier 2s). The Internet was thousand of networks forming a joint whole and everyone was or could be interconnected to peer and exchange traffic more or less equally.

This classic Internet started to change or unravel in the period leading up to 2007, when an ATLAS Internet Study ("ATLAS Internet Observatory, 2009 Annual Report" (1) study found that whereas in 2007, "thousands of ASNs contributed 50% of content, in 2009, 150 ASNs contribute 50% of all Internet traffic" (an ASN is autonomous system or routing domain for a set of network domains controlled by a network operator or service provider).

This meant that by 2007 a mere 150 network operators or service providers controlled or originated over 50% of the global Internet traffic, and that the classic Internet as a series of interconnected peer networks was gone.  These 150 or so dominant network operators wasn't just tier 1 Internet providers and international telecoms operators any longer, but various US Internet web site giants, traffic aggregators, advertisers and distributors like Google, Comcast, Limelight, Akamai and, at the time, P2P-traffic.

As an example, Akamai, the worlds largest CDN operator with some 150,000 servers in 92 countries within over 1,200 networks, now in 2014 claims to deliver between 15-30% of all Web traffic worldwide (what is web traffic in this statement? Http-based for sure, but most likely http-based website traffic only and not adaptive streaming over the http protocol).

In a 2011 follow-up study to the 2007 ATLAS Internet Observatory report named "Internet Traffic Evolution 2007 - 2011" (2) by Craig Labovitz, Mr Labovitz showed that by 2011 some top 10 network operators and Internet service providers now alone accounted for close to 40% of global Internet traffic, clearly showing that large scale traffic concentration and aggregation was in full swing on the global Internet by various Internet giants. These Internet giants and their audience was also mostly American, with the US "growing in both absolute traffic volume and as a weighted average percentage of Internet traffic (grew from 40% to 50% by average aggregate traffic volume in 2011).

So in 2007 some 150 operators of various size and form originated some 50% of the Internet in, while by 2011, it took only 10 operators to control close to 40%.

So one can safely conclude that the Internet by now is dominated by some large Internet giants. Who, it turns out, all have worldwide data centers they operate themselves and all rely on content distribution network operators or infrastructures residing increasingly inside local, last-mile access network operators networks.

This post isn't about the build and reliance upon large, distributed data centers and CDN infrastructures for the Internet parties dominating Internet traffic volumes, but rather that these build-outs can be viewed as a way to preposition and enhance the reachability of services and content by the Internet giants. And as a way to make sure that users get their content, adverts and services with far greater quality of experience than plain Internet delivery from some far-away host. And it would be a far stretch to call the traffic managed and shaped, manipulated, re-routed, load balanced, proxied, pre-positioned or cached by terminal type, location, time of day, web site type etc by these players this way for neutral in delivery or reception.

As noted above, Akamai CDN clusters are inside some 1200 local and global networks, with Akamai URLs showing up lots of places inside Facebook pages, as well as Microsoft and Apple services for instance.  In 2, it's shown that "as of February 2010, more than 60% of Google traffic does not use transit" pointing to the number of ISP deployments (in over 100 countries) and effectiveness of the Google Global Cache Platform (claimed hit rate at 70-990% for local cache delivery) for YouTube and other Google domains.  These distributed cache and CDN deployments and their associated, integrated client-side players, can perform most of the network functionality and traffic shaping as mentioned in previous paragraph.

Since 2011 Netflix has become, of course, a significant source of traffic in the markets they are present in, making up close to 34% of all broadband traffic in the US for instance. And Netflix also relies on a CDN infrastructure of their own design to carry and distribute Netflix movies and vides, the Netflix Open Connect CDN, that is offered for free (or "free") to regional network operators and ISPs to locate inside their network.  As with other CDNs, Netflix Open Connect is meant to save money (for Netflix, for intermediate networks, for access networks) - and to increase the customer user experience, i.e. QoE, for the end-user.  Minimizing video buffering, video load times and achieving best videorate possible towards customers equals increased video rentals per month and higher ARPUs.

It's interesting to parallel the development of Internet giants, their distributed data centers worldwide and reliance of CDNs to distribute traffic and content for maximum quality of experience and performance with two other Internet phenomena developing in the 2007 - 2011 period, namely Internet network neutrality and cloud computing (and cloud-based networking coming out of this latter one).

Since US telco CEO Edward Whitacre uttered the famous "not for free on my pipes" in 2005 (see note a), bringing the earlier work on network neutrality of Tim Wu to the fore ("Network Neutrality, Broadband Discrimination", 2003), in turn strongly influencing and leading up to the 2010 passage of  the US Net Neutrality rule, the real action or infrastructure investments on the Internet, one could claim, took place elsewhere.

Or to re-phrase: While the public debate was quite concerned about perceived non- network neutral activities and shenanigans taking place with local ISPs and telco's, the Internet giants was busy building de facto quality of service traffic management and content manipulation capabilities behind the scenes, that in turn was very much focused on bringing the user a vastly improved Quality of Experience and performance compared to what the plain Internet would enable them to do.

The network neutrality debate boiled down to, I would claim, that all traffic delivery was to be treated equal, neutral and dumb ("neutral" has a better ring to it than dumb, but the effect was the same...) for all users and originators, and without traffic shaping, differentiating quality of service towards the user and without any service discrimination.  So-called last-mile Internet access providers were seen as the main obstacle to or most likely culprits to screw up Internet network neutrality, and much of the network neutrality debate was held I would claim on the assumption that your local telco would be on the opposite end of network neutrality if they could have their own ways.

This level playing field approach was very much in the interest of first-mile and middle-mile operators like the aforementioned Internet giants, that didn't have, usually, last-mile Internet access networks of their own, and were relying on a range of 3rd party local, access providers to bring their stuff and content to the end-users.  For them having one dumb or standard tcp/ip pipe to the end-user would be vastly preferable to having local QoS access schemas all over the place from local besserwissers.

So mini summary - having a local access schema across the Internet that could be treated as dumb and plain as possible without any local or per provider QoS traffic classification was and is a great benefit if you are looking to distribute your content and traffic as uniformly as possible.  Any need, or the real need you have, for traffic management, shaping and positioning can be achieved by having

1. distributed DCs and CDNs as close to the main volume markets as possible, 
2. controlling the end-user client with your own player or portal,
3. logging the client and user behaviour and usage patterns into your own big-data stores to review and optimize service delivery.

Since 2007-2008 we have also witnessed the explosive growth of cloud services for anywhere, anytime and as-a Service IT service delivery.  The virtualisation of servers and compute power gave way to radical economics of scale for the initial set of Virtual Server Providers (VPS-provider), some of whom evolved onto true cloud IT service providers when they added improved self-serve configuration support, pay as you go billing models, ubiquitous network access and so-called location transparent resource pooling.  Soonish a range of network services and functions was also offered on top of virtual servers and not just on dedicated network hardware any longer, paving the way for network virtualization of network functionality.

If one wants to have a look at the future of the Internet infrastructure and networking, there are 2 key developments that should be studied and understood:

1. The development of network virtualization via (increasingly standardized) network functionality virtualization (NFV) and software defined networks (SDN) for network control, inside clouds, backbones or data centers, and
2. Internet bypass and private nets developments

The virtualization of network functionality with NFV for IP routing, caching, proxy, firewalls, NAT and traffic shaping as well as application acceleration will come to mean just as much for the continued development of IP and Internet networking as server virtualization has come to dictate and govern IT-infrastructure and cloud-based IT-production. Network virtualization will over time remove the need for a series of dedicated networking hardware on local customer accesses (corp market) or in the data center and will move advanced traffic management and networking functionality in to the cloud that in turn will lower networking costs for both the corporate and residential market.

And not the least, lower the threshold for giving companies and end-users access to new and advanced network functionality for extended traffic manipulation, shaping and acceleration for instance per application type, IP ports, terminal types, time of day parameters, traffic tariffs and more.

In light of current network neutrality approaches and "all users and all traffic shall be treated equally and without discrimination" one can argue that the whole purpose of network virtualization, NFV (and SDN) is exactly to enable and perform broad, active and wide-ranging network traffic flow re-arrangement, traffic manipulation of all sorts and application discrimination (positive or negative). And whether one wants it or not.  For instance the Google Andromeda introduction this spring, the codename for Google’s network virtualization stack, it was also announced that "Customers in these zones will automatically see major performance gains in throughput over our already fast network connections", resulting from applying "distributed denial of service (DDoS) protection, transparent service load balancing, access control lists, and firewalls". With more enhancements coming...Who says no to more performance?

With websites, Internet services and apps increasingly residing inside one or several clouds, another side to moving traffic for access to the same websites, apps, Internet services and of course SaaS-based applications out of the basic Internet and into the cloud-providers cloud fabrics, their own fiber-nets and data centers, the usual "level" playing and delivery mechanisms of the Internet, with various degrees of assumed network neutrality, no longer applies.

Once the traffic is inside a cloud-providers network and cloud service, the network access and delivery is more akin to an Internet overlay or private, virtual network domain. Most of the larger cloud providers now provide their own direct, private (fiber) access to their own cloud services from major Internet peering exchanges, data centers or telco backbones (for instance Amazon DirectRoute, MS Azure ExpressRoute, IBM Softlayer Direct Link etc) with accompanying "better than Internet SLAs" and support for QoS-tagging of traffic to/from the cloud service in question.

The use of CDN-networks was far along Internet overlay and bypass and this trend will accelerate even more with the introduction of private, dedicated cloud access, that in turn undermines or bypasses traditional net-neutral traffic delivery and traffic management over the Internet:  So far there has been no calls for network neutrality and "neutral" traffic management to take place to/from and inside cloud networks.  Cloud networking policy, management and performance levels are governed by the cloud-provider only.

The Internet bypass phenomena is set to grow, and already a number of companies are positioning themselves as bypass-only providers, for instance International Internet Exchange (see also this article).

Compared to traditional Internet peering and transit it's also interesting to observe the pricing model that most cloud providers utilize for ingress and egress traffic to and from a cloud service: It costs close to nothing or it's free to send data and traffic into or across the cloud-providers infrastructure, while all exit traffic from the same cloud, for intance to the Internet, other network servcie providers or to other clouds, has an associated cost and metered delivery (i.e. cost of GB transfer volumes per month). Not many ISPs would have gotten away with such an pricing policy, and schema and it runs contrary to traditional Internet peering models, and as the line between cloud-providers and cloud-provuders will get blurred, it may be seen as a discrimination of business and market terms between cloud providers and ISPs.  ISPs should rename themselves cloud-provider something and charge for all exit traffic to customers and peers!

In summary it doesn't take a crystal ball to predict that the Internet giants using CDN networks and cloud delivery will only become bigger and increase their share of Internet traffic even more over the next, coming years, while amount of traffic generated by or originating with ISPs and telcos will decrease.  And in parallel, that cloud-based service and traffic delivery will increase using large-scale network virtualization and traffic manipulation, with the traditional network neutrality policies most likely not making it into the clouds at all.

Note a): "Google, MS, Vonage... Now what they would like to do is use my pipes free, but I ain't going to let them do that because we have spent this capital and we have to have a return on it. ... Why should they be allowed to use my pipes? The Internet can't be free in that sense, because we and the cable companies have made an investment and for a Google or Yahoo! or Vonage or anybody to expect to use these pipes [for] free is nuts!".

Erik Jensen,
28.09.2014

Rise and fall of (mobile) Backend as a Service?

Question mark is warranted I think, but this readwrite.com article, "How Mobile Cloud Services Will Consolidate After The Death Of StackMob" (i.e. after being acquired by and folded into PayPal) highlights the gauntlet many VC-funded, narrowly, or is that sharply?, focused upstarts faces.

I wrote about some of these mobile Backend as a Service start-ups in November ("Backend as a Service (BaaS) for mobile services, Internet of Things and devices"), thinking they would be a great match for mobile apps and services in general, for IoT specifically, but Internet giants saw them as a nice addition and as a way to strengthen their own backends (Facebook acquiring Parse etc).

Added challenge was having developers as your core customer group - highly challenging customer group, but noise-value ratio in no way linked to willingness to pay.

Last stand?  - Dedicated Kinvey.

Going to the Mobile World Congress next week, it will be interesting to see if there are movement in this area on the GSMA side of things.


Erik Jensen, 18.2.2014

Large-scale NSA Internet surveillance and bulk traffic collection as an IT challenge

Saturday's article in the New York Times about the low-cost, readily available tools that Snowden used to index and collect NSA documents inside the NSA ("Snowden Used Low-Cost Tool to Best N.S.A."), also points to how large-scale surveillance and traffic monitoring can be done today using low-cost, freely available commercial or open-source software, and commercial Internet or cloud infrastructure services.

In many ways NSA faces the same IT challenges as most large corporations, not the ideal place to be the CTO or CIO probably (or in charge of internal IT security...), but if looking at NSA surveillance and Internet traffic collection as an IT challenge, what are some of the available options?

Most of the following involves some, or a fair bit!, of speculation and what-if-maybe for "Large-scale traffic collection for Dummies", but most of it should be fairly evident if one has followed the Snowden revelations over time.  So how could one go about doing large-scale Internet traffic collection and bulk usage logging?

Firstly, if one remembers the old Roman saying "who watches the watch-guards" (or, ""Who will guard the guards themselves?"), it looks like the NSA should have implemented, across all locations and stations, with contractors and internally, a form of change management and tracking system to track excessive document indexing and downloading, even if it's done with different system admin accounts.  Apparently PRISM could have had some use on the NSA LAN...

Secondly, while this isn't a low-cost option, looking at a map of worldwide submarine fiber cables (for instance http://www.submarinecablemap.com/), there isn't that many key submarine cable landing areas in the US, Europe, Asia and Africa one needs to tap into, splice and copy to gain full access to nearly all Internet traffic by most network operators, telcos and ISPs, with traffic encryption so far seemingly more of a nuisance than a real obstacle to reading traffic and content over this cable.  Fiber splicing by uboat seems like an approach here - i.e. with something like the USS Jimmy Carter.

Thirdly, one odd thing about the XKeyscore revelations last year was the low numbers of servers the NSA employed worldwide, even way back in 2008, to index and/or collect Internet traffic, content and meta-data; only some 700 servers in 150 or so locations.  And able to hold traffic collections only for some days, meta-data for 4 weeks or so.

Already in 2008 of course there were many large scale CDN deployments on the Internet by different commercial operators, having web cache servers deployed in DCs near the major Internet exchanges or with local ISPs worldwide, and directly in the pathway of end-users Internet access and service traversals. Tapping into and using mirrored cached content of 3rd party, commercial CDN services and operators that already then held the majority of users Internet service and content consumption, seems like a way easier and more scalable approach for traffic collection than deploying and maintaining log servers of one own.

Incidentally, that would also give the NSA, as well as the content originators and service providers the opportunity, to deny that they are accessing operator so and so servers directly as has been done - they are working on an mirrored, off-site copy of the servers in question.

Since then cloud computing has come to life in a significant way, and easy to rent IaaS compute, storage and networking resources in almost any country, making it even easier to install and manage XKeyscore related SW and indexing on VMs in country of choice (not that I'm an expert on IaaS-availability in Africa!).  But main thing is - instead of installing and running a battery of servers worldwide for traffic capture and content indexing, it's easier to acquire cache copies of network traffic and content from CDN-operators and/or rent IaaS compute resources worldwide that in most cases are located in most end-users pathways. Or, put differently (slide 20 in the "An NSA Big Graph experiment" presentation: "Cloud architectures can cope with graphs at Big Data scales".

This move to the cloud was also highlighted in the 2011 Defence News pre-Snowden article "Securing the cloud" - highlights: "The National Security Agency has been working on a secure version of the cloud since late 2007. By the end of the year, NSA plans to move all its databases into a cloud architecture while retaining its old-fashioned servers for some time.  "Eventually, we'll terminate the other data base structures," said NSA Director Army Gen. Keith Alexander...  The intelligence community proposal would expand the cloud approach across the community to take advantage of the cloud storage and access methods pioneered in the private sector... "I went in [and] talked to our folks who are on the offensive side" and asked them what would make a network "most difficult" to crack, he said. "And the answer was, going virtual and the cloud technology". See also this one.

The fourth measure to collect user traffic and meta-data is more on the speculative end of things, but since 2008 we have seen the birth of mobile app-stores and mobile users downloading apps for their smartphones in the millions - almost daily.  Instead of creating highly secure and zero-day malware exploits, why not create an app, or at least at app development library, that everyone wants to use, and that people download and interact with daily. And get the app to collect location data, address book access and export, message store access and export etc?  Much easier than doing obscure malware developments and then trying to get the SW-package into peoples devices.

An extension of this is to listen in on leaky apps and the ad networks that most developers of free apps use to place ads in the app and get some sort of kick-back for app generated ad streams.  See "NSA using 'leaky apps' like Angry Birds, Google Maps to siphon user data" for an example.

Moving on to a possible 5th measure or IT approach to do large-scale user tracking and surveilance, Internet traffic collection, we have another area that has "exploded" since 2008, and that is the low-threshold availability of big data log collection systems, distributed storage & processing systems like Hadoop and data analytics tools for laymen (i.e. business analytics).

Once an fairly obscure area of IT administration, who reads and understand router and server OS syslogs anyhow?, analysis and visualization of systems logs turned out to provide valuable business information - usage patterns, performance issues, faulty applications, break-in attempts, what not, system or business wide logging and analysis even more so.  Commercial software coming from these IT amin areas and developments have been identified in the NSA leaks by Snowden, and are now firmly established in the Big Data business.

Large scale storage and processing, or Hadoop by most accounts nowadays.  Grapewine says 9 out of 10 shops looking at Hadoop doesn't actually need Hadoop - but it's Bg Data!, and looks good on the CV - with NSA data collection and processing the one shop being the one that actually could put it to good use (with a Splunk connector or two to get data into the Hadoop cluster, but seems getting data sources for NSA Hadoop clusters isn't the main challenge...).

NSA activities with Hadoop and related technologies are well documented, see for instance

• Defence News article above: ""We started with Hadoop, open source, and we expanded it, added in security layers that we felt [were] needed, and we tested those," he said. "We also added in identity management, data tagging, and the things that we needed to ensure that was secure."), 
• "NSA Using Cloud Model For Intelligence Sharing: New system will run the Hadoop file system on commodity servers and include search, discovery, and analysis tools", 
• Submitting Hadoop software to the Apache open-source se-up: "NSA Submits Open Source, Secure Database To Apache" and ths one.
• NSA Concedes Hadoop Beats Its Pricey Alternatives

OK, several more main street IT services and options still to go, but will come back to that in a later posting.

Erik Jensen, 09.02.2014

Internet traffic as cost driver for telcos - not

One of the longer-running debates in the Internet infrastructure area is the cost impact of Internet services and content delivery by 3rd party providers on local ISPs and telco's IP access network and peering points. Or, taking the opposite view, how 3rd party services and content delivery from the likes of Google/YouTube, Netflix and for instance via the Akamai CDN, has been the key driver helping local ISPs and telcos in actually selling and getting traction for their Internet access services and bundles.  No attractive Internet services and content over the Internet, no local uptake in telco broadband offerings and Internet access business to the residential and business market...

I won't reference the main cases and "conflicts" going on in this area, as I'm sure readers of this blog are mostly familiar with the YouTube and Netflix peering and transit discussions with local or regional ISPs, but rather look into the argument that network build and provisioning of network capacity to 3rd party Internet services and content delivery (that is, 3rd party to the local ISP and telco) are a main or large cost driver to the local ISP or telco.

Firstly, a very rough guide to Internet services or content delivery, using the first, middle and last mile analogy:

1. First mile Internet services and content delivery: All things involved with service and content design, build and preparation for delivery, storing and hosting the service or content in one or several data centers, one or several clouds, for service and content origin.  Internet access capacity from the hoster on to the Internet for service and content distribution.
• Many ISPs and telcos are in the data center or hosting business, so this a source of income for many ISPs and telcos.
  
2. Middle mile service and content delivery: All things to do Internet/ISP traffic exchange (free peering, paid peering, paid transit) and service/content delivery via one or several Content Distribution Networks (CDNs) or via cloud providers. Handover to local access network for end-users, operated by local ISP or telco.
• As above, many ISPs and telcos are in the Internet peering (achieving Internet traffic aggregation for improved transit position towards peers, not necessarily direct money), transit, CDN and/or cloud business themselves mostly through wholesale set-ups, so this is also a source of income for many ISPs and telcos.
  
3. Last-mile service or content delivery: All things to do with delivery of services and content to the local end-user, by fixed or mobile broadband.  Will in many cases be done via a mix of locally deployed 3rd party CDNs or ISP caches, a mix of layer 3-5 load balancing by 3rd party or ISP.  And for content provisioning, adaptive streaming taking end-user bandwidth and client processing capability into the equation (negotiated with content origin or CDN edge caches in question).
• Main source of income for most ISPs and telcos, based on selling local access bundles, 3rd party service add-ons and ISP bundled services like TV or IP in various forms, music services, sponsored tablets and smartphones tied to broadband subscription of some sort and voice over IP services. 

Looking at this, it's possible to make the argument that ISPs are uniquely positioned to cash in on and control the distribution of 3rd party services and content deliveries )(overlooking that Internet majors seldom hosts with local heros that much), but still there is dissatisfaction with how 3rd party service providers and content services like Netflix and YouTube wrecks ISP and telco backbones and economies. Why is that?

Looking at ISP cost categories and drivers, they can be divided as follows (not necessarily in order of cost impact):

1. FTEs and man-hours: Skilled personnel needed to design, deploy and operate quite complex network structures in the backbone and local access areas, protect from Internet hacking and DDoS attacks.
2. Network equipment: Routers, switches, firewalls, control planes, backbone fiber, local access network, POP and (over-)capacities at major national and international peering and transit locations, DCs and co-lo space for network equipment.
   
3. IT IS and back-office IT systems for service authentication, service provisioning, billing and help-desk.  A major cost driver for most telcos and ISPs, as service complexity, number of integration points, legacy IT IS systems and integration of virtual network functions seems to multiply out of control weekly.
   
4. Internet transit and peering capacity, off-net capacity: ISPS and telcos needs to buy Internet CDN, peering and/or transit network capacity with or from 3rd parties. This is the cost element that is most often brought forward by ISPs and telcos in the "OTT is eating my network" debates and arguments.

Looking at these costs elements and drivers, one can argue that

1. FTEs and man-hours per ce are under control, and going down due to outsourcing of many network operations tasks. Key challenge seems to be number of FTEs needed for a certain operational capability, not the salary levels per FTE in itself.
   
2. Network equipment: Overall trend is for this kind of equipment following Moore's law, giving ISPs and telcos ever more processing power or networking capacity. Same downward price points for DCs and co-lo.
   
3. Internet peering/transit and CDN price points are involved in a race to the bottom giving Mbps or that GB/month pricing that would have been unthinkable some few years ago.  Although a bit old, have a look at "Internet Transit Prices - Historical and Projected" and for CDN pricing, "CDN Pricing Stable: Survey Data Shows Pricing Down 15% This year"

This leaves IT IS and back-office IT systems as the fall-guy, and here the legacy plus complexity spiral has most companies, not only ISPs and telcos, but banks, insurance, governments and institutions trapped.  IT IS remains a key cost driver for most companies, whereas Internet giants like Amazon, Google, Netflix and Facebook has achieved rapid service development, service provisioning and roll-out and consistent service performance (equals commercial success) precisely because they have managed to get their IT IS systems under control or rather, into a flexible and adaptable software development process environment being at the forefront of cloud developments. And IT IS then becomes a competitive and cool tool rather than some boring way in the back back-office legacy and career ending thing.

An additional argument that can be made is also the overall cost savings ISPs and telcos have been able to achieve using Internet and tcp/ip technologies (IP on everything, everything on IP...), and not things like pre-Internet ISDN and ATM, OSI-stacks, for integrated services networks and one network for multiple services that we all take for granted today. Imagine a competitive IT IS stack for OSI on ATM versus one control plane for IP-based networking.

In summary: ISPs and telcos stand to gain the most from adopting SW development processes, environments and cloud technologies on par with Internet giants to simplify their IT IS stack, speed up service delivery and competitiveness.  Not being competitive in this area is a far greater threat than Internet 3rd parties consuming too much bandwidth without paying for it.

Erik Jensen, 27.1.2014

The net neutrality that never was: Bringing net neutrality up to speed

Lot of comments and writings has been made after the US Court of Appeals for the District of Columbia Circuit last week halted the FCC's "ban on traffic blocking and discrimination by Internet service providers because the FCC had not designated ISPs as common carriers" (*). Most of the writing has fallen into the camp that this more or less means that non-discrimination of Internet traffic and net neutrality for Internet traffic and parties is dead and that this signals the end of basic Internet traffic delivery freedoms. In the US. And seemingly most other places as well.

In the US, were most areas and cities are only served by one dominant service provider or telco, the situation in many ways are more dire than most other well-functioning markets, where there are multiple Internet access providers and access technologies to choose from, and one dominant access provider can't run amok in how basic access services are provisioned.  And, let's say, play favoritism with one's own services or content offerings.

That said, I think the basic Internet neutrality approach and arguments has been flawed from day one, or at least half of what net neutrality implicates.  To me, net neutrality could and does entail two things

1. Neutrality as to how network protocols, services, applications and devices are treated, i.e the same and without prejudice or helping hands of any sort
   
2. Neutrality as to how communicating parties are treated, i.e. basically how sender and receivers or peers in a networking session are treated, without prejudice and equally.  For instance
• Broadband Internet access customers treated the same for the same service or session
• Content providers, Internet service providers treated the same towards broadband subscribers and in IP backbone management

Number 2 is taken for granted by most parties I believe, whereas number 1 I think was and is a failed approach as most network protocols, Internet services and applications as well as end-user devices never was or is designed to be "neutral" or equal to one another, nor to they behave particularly "net neutral".

A couple of examples as to why "network protocols, Internet services and applications as well as end-user devices" are closer to the "all you can eat" camp or passive resources, than nicely behaved, neutral net citizens:

1. Some layer 5 tcp/ip protocols are better designed or designed for better performance than others (depending on how old they are, what they are designed or forced to support or utilize the underlying tcp/ip stack).  Some tcp/ip protocols are also better at utilizing or able to use sliding window mechanisms, giving them larger transmissions windows and better abilities for avoiding IP traffic congestion.  And there are of course performance and behaviour differences in how he basic tcp/ip stack is designed and implemented in differents operating systems and network elements.
   
2. Use of http for adaptive media streaming means streaming clients will quite aggressively seek to reach the highest encoded video rate achievable for their broadband connection and media device playback capability, meaning other network protocols and service deliveries as well as non-adaptive devices will suffer and be put to the back on the broadband connection.
   
3. There are few or none IP packets that traverses the Internet between a sender and a receiver or between two peers today that hasn't been modified in some way by one or more load-balancers, NATs, layer 3-5 traffic accelerators, a cache server or CDN service or had their DNS/IP header re-written or payload compressed in some way or modified depending on OS, browser, location or time of day, meaning that full-time network modification or alterations to Internet traffic is already a fact of life.  And was also some 6-7 years ago. MOst of these are very useful and necessary developments for IP and Internet traffic management, scaling and optimization as the basic tcp/ip protocol stack is old.  Http 1.1 is also getting geriatric. Meaning layer 3-5 protocol optimization, caching and application level load balancing are useful and necessary additions to basic, "neutral" IP networking.
   
4. Most Internet traffic and content today are served by a couple of Internet giants.  As measured in the ATLAS Internet Observatory 2009 Annual Report, "In 2009, 150 ASNs contribute 50% of all Internet traffic".  Would anyone be surprised if some 25-50 ASNs are behind closer to 75% of Internet traffic in 2014?  These Internet giants are using their own fiber backbones and cache/CDN/cloud infrastructures for service and content delivery towards end users, not the basic Internet itself, and Internet bypass for first-mile and middle-mile service delivery has been the default approach and get things done in large-scale settings for years.

These are some of the ways basic Internet network protocols and service delivery never was or isn't "neutral" from the outset, and with cloud-based networking and service delivery becoming the norm towards broadband users and customer in both the business and residential market, this "non-Internet" approach will only accelerate in the coming years.  

As noted above, many of the advance in layer 3-5 traffic optimization, acceleration and load balancing greatly helps in the delivery of services and content towards end users.  And I would like to see the user who takes his YouTube videos or mobile app usage without any traffic assistance at all and opts for a "neutral" non-QoS delivery (it's not even best-effort) over a optimized and accelerated service delivery.  

For net neutrality I think the main focus for policy development should center around 

• Treating everyone the same, including giving everyone the option of paying for optimized transport and Internet traffic management on Internet first-mile, middle-mile and/or last-mile sections
• Open and transparent information as to how first-mile, middle-mile and last-mile service providers (including CDN- and cloud operators) utilize Internet traffic management and optimization technologies for their networks, CDNs or clouds and what's available for 3rd parties to utilize for their service delivery through open APIs, SDN interfaces or manually.  In short, what's their internal and 3rd party QoS regime and policies.
• Don't limit IP network, Internet and IP traffic management developments by enforcing a net neutral straitjacket on Internet networking and service delivery that hasn't kept up with reality for the last 10 years or so.

In short: The net was never neutral, but users on the Internet needs to be treated neutrally.

Erik Jensen, 20.1.2014

Why doesn't mobile Internet access translate to mobile revenue and increased ARPU?

Mobile Internet access and traffic - there's no shortage of it.  In fact, in the latest Sandvine Global Internet Phenomena Report for 2H, 2013, mobile traffic is shown to grow rapidly for every market being monitored.  For instance:

• North America, since 1H report: Mean monthly usage has made a 13.5% jump, increasing from 390.1 MB to 443.5 MB
• Europe: Mean monthly usage has increased 15% from 311 MB to 358.4 MB per month
• Asia: Increasing from 700.4 MB to over 1.1 GB per month

Overall, mobile traffic (i.e. from mobile devices) only makes up some 20% of Internet traffic (see KPCB 2013 Internet Trends, slide 32), but growing at 1,5x per year or more, mobile broadband traffic is on a pretty good trajectory.

Similarly, the Ericsson Mobility report for November 2013 states that "the increase of monthly mobile data traffic in Q3 2013 exceeded total monthly mobile data traffic in Q4 2009" and that there was a "80% growth in data traffic between Q3 2012 and Q3 2013".  And looking forward, "mobile data traffic is expected to grow at a CAGR of around 45 percent (2013-2019) leading to a 10x growth in mobile data traffic between 2013 and 2019".

So, volume-wise, mobile traffic numbers look good if you are in the mobile broadband area or a mobile access provider.  Does the traffic increases translate to money for mobile access operators?

While mobile traffic is increasing significantly, ARPU for mobile operators, for both mixed voice and mobile broadband providers as well as voice only providers,  are expected to fall in the next years (GSMA and OVUM numbers a bit old):

• GSMA: "European mobile ARPU falls 20%", 1, 2
• Global Mobile Outlook, OVUM (2011): Monthly ARPU remains in steady decline across all the regions, and we expect it to fall at a CAGR of 4% from 2011 to 2016.
• The Mobile Economy 2013, AT Kearney: Despite the growth in usage of voice and SMS and increasing numbers of data subscriptions, ARPU rates have declined across every region globally. The overall global ARPU rate has fallen by 7.6% p.a. from US$19 to US$14 per month, with the highest reductions in 2010-2012 seen in Africa (-10% p.a.) and Europe (-7% p.a.)

So, revenue per user going down for mobile operators, but average traffic volumes per subscriber going up rapidly.  Why?  There's a number of reasons among them:

1. Despite Moore's Law and networking units getting ever more capacity at ever lower prices, functionality, complexity, service management levels and operations transactions per networking unit is going up.  There is seldom less complexity introduced with new mobile G generations and always-on mobile units, costs for building and maintaining mobile networks are also increasing.
2. Race to the bottom.lack of service differentiation: Telco and mobile operations has squarely entered the mass-production or utility service era, and there is little noticeable service quality or service range differentiation once basic mobile coverage has been ensured. Operators are competing on level of mobile handset subsidies and traffic volume/capping bundles.
3. Mobile access subscription and service does not translate to any significant degree of share of "mobile wallet and spend" - customers get their mobile service needs by Internet and 3rd party providers.
4. Lack of investment or market foothold in first-mile services (i.e. hosting, extended comms and messaging) and middle-mile infrastructure and services (i.e. CDN, on-net traffic aggregation) for mobile operators and telco's .

Point 4 can be illustrated by how things are looking in the mobile ad space and mobile app & service subscription space - much better than mobile voice and broadband ARPU developments! Some indicators:

1. Business insider,THE FUTURE OF DIGITAL, 2013: Mobile is the only media time that's growing (slide 24), Mobile is now approx 20% of e-commerce traffic (slide 37)
2. Mobile ads now close to half of Facebook ad ARPU - slide 37 in the KPCB report referenced above
3. Twitter: " More than 70 percent of advertising revenue came from mobile devices in the third quarter, compared with 65 percent in the second quarter." (2013 numbers, of total Twitter revenue of $168.6 million in the last period).
4. Gartner is predicting that worldwide revenue from app stores will increase this year (2013) by 62%, bringing the total industry revenue to $25 billion dollars.
5. Business Insider, "The Results So Far From Holiday Shopping Point To Huge Gains For Mobile Commerce This Year":  ...mobile commerce grew more than twice as quickly (as e-commerce overall), at 63%, and accounted for nearly $940 million in sales on those three days. ...one in four e-commerce dollars spent on Black Friday and Thanksgiving were on purchases made through mobile devices. ...Tablets have emerged as a principal engine for mobile commerce growth.

So, to answer the question raised in the title: It turns out there is no direct relation, so far?, between providing mobile Internet access and capturing share of mobile traffic and services revenue, or share of mobile customers spend on e-commerce, Internet services and use of traffic-intensive deliveries.  Because with whom customers choose to have their mobile access isn't the same parties that they choose to use for e-commerce, Internet services like messaging, social media sharing, file sync, photo storage and get delivery of movies and entertainment. 

Erik Jensen, 15.1.2014

Cloud or not, compute or not, Internet or not.

Biggest lesson from the Snowdon revelations: Social engineering and employee "activities" of various sorts is still the greatest threat to company IT security. As it always has been.

All the worlds best, strongest and well maintained network and IT security systems won't matter if it's being bypassed or circumvented by social engineering, employees or 3rd party, on-site contractors, and that holds equally well for on-site as well as cloud IT services.

In many ways the debate nowadays if cloud IT is secure or not, if a company should be making the jump or not, or maybe even pull back the initial services and data that might have done the cloud leap, resembles the discussions at CxO level 15-20 years ago whether a company should hook up to the Internet or not (beyond the IT departement and some islands of deployments):  It wasn't perceived as secure, there were strange things out there and hackers might get into the LAN!

Most or nearly 100% of companies went with the Internet and gaining all that now is taken for granted re Internet, web and seamless communications services after putting in something resembling a "Internet access and usage security" policy and a big firewall.  Users were allowed to browse the Internet.

20 years later and many line of business units or departments has not only browsed but also put to good use a number of SaaS cloud services as well. In many cases the new IT services buyers in the HR, marketing or finances departement not knowing that the solution they had to have, and ASAP!, were in fact a cloud service delivery.  Lately the IT departement has come around to the cloud IT service delivery model as well.  And looks likely to continue on that path, as the business benefits are seen to outweigh the NSA or governmental listening in drawbacks (were take for granted anyway, just as for general Internet traffic and service usage by most pragmatic companies and network security managers).

2 additional factors are that if you aren't doing cloud computing or service delivery, someone else certainly will still be doing it and increasingly getting better at it, while you risk being stuck with on-site service model and lead times.  And, secondly, it's not as if having everything behind that big firewall doesn't carry some risks as well, be it on human, social or system level...

This pragmatic and take appropriate measures approach were highlighted in a recent IDC survey, "2013 U.S. Cloud Security Survey" (Sep 2013) of  IT executives in North America and Europe, loosely summarized as "yes, there are security and surveillance concerns, but the economic benefits and increased business agility outweighs security concerns".

What are the measures that can be taken by most companies to overcome cloud security concerns and issues?

A new infographic by Sage highlights the first steps that should be taken by anyone, for any IT solution really:

1. Establish the IT and business security policy for IT in general and the the IT solution in question
2. Train your employees.  On the IT policy and the IT solution in question, best practises etc.
3. Assess business needs, i.e. what business data needs to be where, accessible by whom and how. With what kind of service levels.
4. Choose the right supplier and service for number 3

Erik Jensen, 12.12.2013

Cloud IT services for SCRUM, companies getting agile with cloud services

A strange mix or bedfellows maybe?

• SCRUM IT development with cloud IT services
• AGILE approaches and companies and cloud, or rather, how agile companies and organisations are well suited to utilize and benefit from flexible service delivery by way of cloud IT.

Putting it very broadly - agile companies using SCRUM for a stepwise, iterative approach to IT service and business development stands to gain the most from the flexible and on-demand service delivery nature of cloud IT services, be it in the cloud infrastructure area, for platform level kind of services or in the areas of application and ready-to-go line of business services delivery.

Let's say you have agreed upon the goals and visions for your SCRUM-based work effort, the initial roadmap and product backlog has come into place with the product owner, stakeholders and team. And the initial design, sprints and infrastructure requirements has been decided upon. Then it is very easy - and, not the least, very fast to start doing the initial prototyping and spring functionalities on a basic IaaS or PaaS set-up, with limited or as-needed capacities for the core team development and testing than can be put up, expended or thorn down as needed on the hour.

Or, for instance, one is doing a SaaS-area SCRUM development effort, involving migration legacy data onto the new SaaS-services.  Once again it's quite easy to start with some initial SaaS accounts for the main SaaS roles involved, and start  importing the core data sets and look into the core SaaS functionality for the business, data flows and data collections in question.

Or for the "developing functionality from the scratch" kind of SCRUM-runs, start with some basic IaaS capacities or PaaS feature set, and then add storage, processing, networking or database capabilities as the sprints falls into place and the testing, piloting, beta and launch gets underway.

That's the basic lineup, matching the agile and iterative approaches of SCRUM with the same stepwise and flexible capabilities of cloud-based IT services. For companies who have already signed into the agile and lean approaches for business and IT development, this shouldn't be a surprise, and this way is taken for granted - cloud services and agile approaches came about more or less in the same cultural happening or paradigm even.   What's harder to archive is to change or develop let's say traditional businesses and organisations into a agile, iterative and sprint-enabled organisation.  And that's a whole MBA course just there...

Erik Jensen, 4.12.2013

Backend as a Service (BaaS) for mobile services, Internet of Things and devices

Backend as a service (BaaS), is making an appearance as a new "as a Service" cloud or SW development approach to give developers (mostly) a general way or an API into common application enabling infrastructure.  It gives web, mobile app and Internet of Things (IoT) developers a way to link their client services and applications to backend cloud processing and storage, as well as providing generic functionalities like user authentication and management, service management and logging, push notifications and integration with social networking services and more for their apps.

With all this going on with a BaaS, it might be easy think it's a mix of cloud IaaS, PaaS and SaaS geared towards mobile developers and clients, and gives developers a turn-key, if there ever was anything "turn-key", software modules needed to run the general backend services of mobile applications.

Kinvey has a great mobile dev and services ecosystem map, that shows were BaaS is generally positioned and residing.

Why would anyone use a BaaS kind of backend, and not develop their own to suit service specifics?  Main reasons seems to be that BaaS makes it

1. Easier to cover multiple terminals and form factor, different mobile operating systems and for instance multiple authentication, logging and payment schemas.
2. Easier to prototype, quicker to launch and both easier and quicker to scale the backend as an app or mobile service might take off or see seasonal capacity demands
3. Possible to outsource or transfer many of the security issues and worries associated with mobile apps, payment and general hacking.
4. Easier to also cover Internet of Things kind of devices as most of the mobile BaaS players have IoT enablement on their roadmap also.

This is a developing area, and merges and some big names VC investments are certain to come through the next 6-9 months.

Some BaaS players you might want to check out:

1. Firebase
2. Parse, recently bought by Facebook
3. Kinvey
4. Appcelerator
5. StackMob
6. AppliCasa
7. StrongLoop
8. Kumulos
9. BaasBox

Erik Jensen, 25.11.2013

Who are the cloud services buyers?

Already there are some stereotyping on who is buying, i.e. actually paying for, cloud IT services:

• Developers and upstarts buy Platform as a Service type of cloud services, i.e. development environments that can be tailored to meet developers needs or configurable platforms that upstarts can tailor to their needs
• CTOs and IT departments buy cloud infrastructure, Infrastructure as a Service, type of cloud IT services, i.e. compute, storage or networking as a service
• Everyone buys cloud app services and Software as a Service, but usually it's line of business units that buys a specific SaaS-service for their task or project deliverable at hand (and CTO or IT departement are out of the loop)

This is already seeing some change, for instance enterprise IT are increasingly looking at and buying PaaS kind of cloud IT services to cover business needs that can be met using "one size for all" SaaS applications and basic cloud IT infrastructure setups.  And for instance HR or finance departments that wanted that one, great, must-have SaaS application for their line of business, finds out that there are integration issues or data exchange issues once they have 2 or more SaaS apps up and running from 2 or more cloud providers.  Can the IT departement please help sort out this mess?

Bain and Company had a great break-down of cloud buyers in a 2011 report (The five faces of the cloud, by  Michael Heric, Ron Kermisch and Steve Bertrand), listing 5 CIO buyer categories according to cloud "adoption speed and willingness":

• Transformational: These early adopters already use cloud computing heavily to transform business IT and delivery to their business, with on average more than 40 percent of their IT environments relying on one or more cloud models.
• Heterogeneous: These companies are looking to evolve IT service delivery and capabilities and typically have an diverse mix of legacy systems and newer technologies like virtualization and cloud computing. Assumed to make up more than 40% of the buyers in 2013.
• Safety-conscious: Balancing security with growth, these buyers and companies are particularly concerned with the security and reliability of their IT environments. They understand the value that cloud computing offers, but are willing to compromise to ensure that their IT and business environment is safe and secure. Private cloud and hybrid public-private cloud models have the most appeal. Along with the transformational type of companies they are the biggest cloud IT spenders by 2013.
• Price-conscious: Have their TCO for IT services in place years ago, these bottom-line focused companies purchase cloud technologies and services primarily for cost savings and to deliver basic business functionality.
• Slow and Steady: This is by far the largest group of companies and IT buyers (some 44% of companies in 2011) and do not yet appear ready to adopt cloud computing in a progressive way, although they express interest in exploring offerings if a provider can slowly and steadily guide them.

The key thing in the report is the observation that "... early adopters generate ~50% of cloud spending today (i.e. in 2011), but ~90% of growth through 2013 will come from other companies".

Assuming, for many reasons, that Europe and the Nordics lags some 2-3 years behind the US in cloud adoption and uptake, a lot of the stories and hybris we are seeing in the Nordics for cloud take-up and usage today are coming from these transformational and heterogenous early adopters, but the real big money are on hold until the larger companies and enterprise IT starts adopting cloud IT in a forceful way. Which should be 2014 - 2016 in the Nordics approximately.  

A finishing note on the role of the CTO and IT departments, that in many cases are being bypassed by the CFO, CIO or CMO or developers buying cloud services directly themselves - what are they left with?  In many cases being very or too much focused on their on-prem IT platforms and services, they will be forced to take on on-prem and cloud based IT service delivery as well, evolving into an IT broker of physical or virtual application services for their organisation once the CFO or CMO realize that handling 2 or more cloud services and applications aren't that straightforward anyways when it comes to user support, login hazzles, performance variations, billing and security across different cloud providers.

Erik Jensen, 20.11.2013

Internet of Things ecosystems and balkanisation risks

Like Big Data, Internet of Things (or IoT for short) has been talked about for years, and seems on the verge of making it big the next 1-36 months or so.  Just as real-life management intelligence and business value from Big Data logging and analytics.

Just as Big Data solutions and systems has to deal with tons of different/proprietary log formats and data sources within an enterprise or from public data sources on the Internet or other places, then applying application or vendor specific data collection and log normalisation, and doing application specific mapping to business KPIs, reports and analytics, so IoT faces a number of non-standardized or vendor-proprietary challenges to become a true interconnected web of things, things to humans, humans to things etc.

There are numerous non-standardised issues and management in the areas of IoT security (service access to things by other things and humans, authentication and authorisation, management and reporting of denial of service and hi-jacking of devices, device upgrades, logging), identification and naming schemas for things, common IoT metrics, real-time control and communication protocols, subscription models and reporting.

A recent IETF Internet report draft, "Security Considerations in the IP-based Internet of Things,               draft-garcia-core-security-06") seemingly puts a lot of faith in IPv6 and web services in general to facilitate IoT developments ("The introduction of IPv6 and web services as fundamental building blocks for IoT applications [RFC6568] promises to bring a number of basic advantages including: (i) a homogeneous protocol ecosystem that allows simple integration with Internet hosts; (ii) simplified development of very different appliances; (iii) an unified interface for applications, removing the need for application-level proxies."), but also adds "Although the security needs are well-recognized, it is still not fully clear how existing IP-based security protocols can be applied to this new setting".

On a general level this is of course quite all right but of one looks at the developments of for instance, and quite relevant, mobile ecosystems where some key players control their entire ecosystem (clients and device OS, programming APIs and SDKs, backend for authentication and billing, app stores, ad networks integration etc), homogeneous protocol ecosystem for IoT and unified interface for IoT devices, clients and services, looks a long way of.   And so far, in my opinion, most IoT devices and services for home automation and IoT, in-car or transport IoT, M2M payment arrangements and more are proprietary and vendor specific.

For instance for home automation, it's not easy or doable at all to get Belkin WeMo units to talk to or interact with Nest units or Telldus units.  Or reach them through a common programming interface or backend. (although I should backtrack slightly here - the great IFTTT scripting service is starting to emerge as a common way for end-users to program their devices, and is supported bu Belkin WeMo and Philips for their Hue range for instance).

With that backgrounder, are there risks of IoT being balkanized, and that IoT devices and services will become vendor or ecosystem proprietary?  Or are there standardisation efforts underway to overcome this risk and 2-3 vendors dominating this field over time as we have seen in the mobile area, Internet video or social media area for instance.

Currently the IETF doesn't seem to have a RFC track for IoT comms and networking standards, but the IEEE standards organization are now finally are gearing up (or, they had their first IoT report out in 2005), and are meeting for their initial IoT standardisation tracks.  Will probably take some years and in the meantime it's not hard to predict that this developing and promising business area will see most gadget, cloud and Internet OTT players getting involved (why not Facebook for home automation and control, Microsoft Xbox with Kinect for same and as automation hub, Android and Google Glass for an Google approach, Apple TV or iOS-devices for same etc).

And getting involved here means each vendor building and securing their IoT ecosystem on both client and backend/cloud side, extending device OS (iOS, Android) to cover IoT functionality and attract developers and påartners into their IoT ecosystem.  I would put my own money on one or two of this, even though it means IoT balkanisation.

Looking to read up on IoT developments and work? Here are some pointers and vendor samples (in no particular order):

1. Wikipedia on IoT
2. McKinsey Quarterly report, The Internet of Things
3. Dark reading, Identity management in the cloud by Ericka Chickowski
4. IFTTT. And an article on how to get started with IFTTT from ReadWrite
5. OpenIoT - Open Source Solution for the Internet of Things into the Cloud
6. CastleOS for home automation
7. You are most likely a IoT service provider - Google Maps gets real-time traffic, crowdsources Android GPS data
8. Postscapes - tracking the Internet of Things
9. IoT cloud specialist - Arrayent
10. IoT developments environment and tools, IoT cloud - Xively
11. Device relationship and ID management - Forgerock IRM

BTW, what are the Balkans and Balkanisation?


Erik Jensen, 14.11.2013

Cloud security and surveilance - what are the non-US alternatives?

GigaOM is quoting a new survey by PriceWaterhouseCoopers (PWC) released last week, saying that some " 22 percent of German companies now see the risk of using cloud services as “very high,”... 54 percent say risk is high or very high.  ...while 15 percent want to switch to European tech providers that won’t cooperate with American or British intelligence services."

I haven't found the PWC survey on question, but it mirrors findings in the "How Much will PRISM cost the U.S. Cloud Computing Industry?" report from the Information Technology & Innovation Foundation earlier this summer finding that "10 per cent of respondents outside the US had cancelled a cloud project with an American firm because of PRISM, while 56 per cent said they're less likely to use a provider based in the US."

OK, let's say you are tasked with finding a secure cloud provider outside the usual US ones and that doesn't have US offices, subsidiaries or business units that would be covered by FISA/NSA or US National Security Letters that will impact non-US operations or locations as well, and need to come up with a cloud infrastructure provider that covers processing, storage and networking at competitive prices and that have feature-parity more or less with the leading players, i.e. Amazon AWS.

What are the options?

One could start looking at German and Swiss providers that have some track record legally, nationally and culturally for safe-keeping and data privacy.  UK and Swedish ones would be out because of GCHQ and FRA impacts, France with their equal, same thing with Norwegian providers as 99% of Norwegian Internet connectivity goes through Sweden.  One place that's often overlooked is Finland, but they have some players as well.

With that in mind, some cloud infrastructure players that have the basics covered for IaaS, an extended feature set for IaaS and self-serve IaaS at competitive prices. It's not an extensive list and I haven't checked all the way if they have US units or not, that would be impacted by FISA or US National Security Letters. Also, remember, it's very hard at most times to say a cloud provider are Swiss or Finnish or is located in a particular country - many DCs and servers for IaaS might be located in one country, but management and ops are done remotely, Internet infrastructure for the service (DNS, SSL certificates, L3-7 global load balancing, service logging etc) are done from a remote location, that might have data or data control for a remote DC running through them.

Some German cloud infrastructure providers worth a look:

1. Profitbricks: I thought of them from the start, but now see they have a US unit, and they would be covered by FISA or National Security Letters just as any US company.  Still gives a good indication of service and feature levels available from leading European cloud providers.
2. Internet4YOU: Servers, storage and DCs in Germany, covers most IaaS-areas
3. dynaCloud: OnApp based cloud provider. Also CDN-services.
4. The unbelievable Machine Company: Name alone makes them worth a check

Some Swiss cloud providers

1. Exoscale: Cloud infra offering. See also "In Switzerland your data is safe" section.
2. Safe Swiss Cloud: Focus on security and privacy
3. Swisscom dynamic computing: Covers IaaS basic, has online configurator and more.
4. Incloudibly
5. Cloudcom: Cloud servers with DDOS-protection and more

Alternatives in Finland:

1. Tieto cloud services: Also has a Swedish, FRA-impacted counterpart
2. Nebula
3. Hostingservice.fi: Another OnApp based contender

OK, this is by no means a comprehensive list, and a closer review might find that some of these providers do indeed have US affiliates or hosting of some sort, from their own router at US IX or using some back-up facility of sort.  But main thing is that there are lots of alternative cloud providers in the IaaS-space and that one isn't necessarily forced to go with NSA-compatible ones to get business or developer requirements fulfilled.


Erik Jensen, 11.11.2013