What’s Next for the Serverless Platform
In 2015, we released the Serverless Framework, wanting to help developers build applications on top of new serverless computing services, such as AWS Lambda (also known as Functions-as-a-Service).
The Framework has since become one of the fastest growing open-source projects on GitHub and a major focal point of the serverless movement in general. Our project tends to serve as a bellwether: a place where anyone can go to track use-cases and emerging trends.
This gives us some insight into why the serverless movement is so strong and leaves us clues as to where it’s heading.
The Past
When the Framework was first introduced in 2015, we were surprised by how many progressive enterprises lined up to try it out.
These early adopters were true converts: defectors from the world of microservices, containers and traditional cloud architectures. They sought refuge from the complexity of modern software development — the daunting amount of tools and best practices, operational overhead, and general propaganda pertaining to the religious war of “how to build x”.
While the “serverless” term was a bit semantically controversial — and remains so — early adopters haven’t minded. Engineers were desperate to get products to market and reduce overhead. The word “serverless” promised them a technology that just got out of their way. It was a message that resonated.
The serverless community was united under one maxim: less was more. They wanted more results, not more technology.
Convenience Attracts Uses
By the end of 2015, engineers began conducting serverless proofs-of-concept within their organizations. Success in those early trials empowered them to look at every piece of their existing business logic and ask, “Will it Lambda?”
Initial serverless use cases were dominated by data processing. But suddenly, engineers were cramming all types of workloads into Lambda: using serverless compute to build back-ends and automate workflows. They churned out auto-scaling, pay-per-execution microservices with minimal effort or administration. The gain in efficiency was too enticing to ignore.
Even as some early limitations of serverless compute became potential roadblocks, engineers chose to compromise or find workarounds. Cloud providers rushed to address limitations with relentless fixes and new feature releases. This provided, even more, momentum for serverless-first engineers.
To realize the full potential of serverless architectures, the community needs a single, uniform experience for deploying functions and managing their lifecycles across all serverless compute providers.
When the answer to that initial question, “Will it Lambda?”, was no, companies kept pushing forward with a positive outlook. They said, “Okay, how long until it can?”
Serverless computing took the majority of administrative tasks that occupied an engineer’s day and eliminated them. This additional free time began to manifest in new projects and creative tinkering. All sorts of projects surfaced: chatbots, DevOps automation, file manipulators, policy enforcers, webhook listeners, HTML renderers, scheduled tasks, and so much more.
In traditional architectures conferences, talks are often about administration; in serverless conferences, talks are often celebrations of things that were built.
Rise of the Serverless Team
An interesting situation arises after a serverless project is completed: Its engineers are (mostly) free to take on new work.
Within companies, rogue serverless pioneers gained a following and formed the first unofficial serverless teams — teams which were remarkably productive, despite their size. They often operated as independent units, practicing random acts of automation and venturing on missions to tackle new jobs in a serverless fashion.
By the end of 2016, only a year after the Serverless Framework was released, several gung-ho teams in large enterprises had already put hundreds of serverless functions into production.
Serverless First
Managers took notice. Top-level business leadership were ever chasing aggressive digital product goals. They wanted to innovate faster than their competitors and immediately ship to market, rinse, repeat.
Business leaders started looking at serverless architectures for a competitive edge — some CTOs even opting to build their own serverless prototypes for faster buy-in. Large enterprises who had not even embraced public cloud yet wanted to jump straight into serverless.
They viewed it as a far more accessible version of the cloud, one which made it relatively easy to convert their large workforce into a heavy-hitting team of cloud engineers.
Present
The momentum of the serverless movement has inspired other vendors to offer their own serverless compute services. It’s not just AWS Lambda anymore.
There is now Azure Functions, Google Cloud Functions and IBM Cloud Functions (based on OpenWhisk), along with smaller vendors serving specific use-cases, like Twilio Functions, PubNub Functions, Auth0 Webtask. There are also several serverless computing implementations being built with Docker and Kubernetes, like Kubeless and Fission.
Larger providers are introducing dozens of new managed services and APIs, like Google’s Speech and Vision APIs, and AWS’s Lex and Polly services, which pair nicely with serverless computing and generally follow the serverless ethos.
The rate of feature competition by major cloud providers is accelerating. Meanwhile, developers and organizations are delighted by the growing number of tools they have at their disposal to solve more problems and deliver more innovation.
Time to Operationalize
Serverless advocates want to standardize serverless development so others can adopt it throughout their organization.
There is clearly a need for better tooling to support serverless development and lifecycle management. Organizations have adopted many platform-specific solutions for this (such as AWS CodeBuild), but with slight hesitation. Vendor lock-in takes away any flexibility they have in terms of runtime and choosing their own lifecycle management tools. It also makes it hard to shift these provider-specific architectures to another provider.
So for serverless-forward teams, The Serverless Framework has become an integral part of their adoption story.
Incentivizing Multicloud
Somewhere along the way, an unintended consequence of serverless computing emerged. It made leveraging a cloud’s services as easy as uploading a function. Need to adopt a cloud provider? Stick a function there. Further, serverless functions were auto-scaling and pay-per-execution, which meant having functions provisioned across multiple regions, and even multiple providers, was trivial.
AWS Lambda was originally thought of as a way to glue AWS’s services together. It quickly became clear that serverless computing, in general, was a way to glue all services and platforms together. This binding power offers an interesting take on multi-cloud and hybrid-cloud architectures.
While exciting, it does prompt a discussion of what the serverless architecture should look like. Should we continue to think of serverless architectures in a platform-specific manner, given that they live everywhere all at once? Further, if serverless architectures are universal, what does that mean for data?
Future
There is currently no standardization across vendors. To realize the full potential of serverless architectures, the community needs a single, uniform experience for deploying functions and managing their lifecycles across all serverless compute providers.
The Serverless Framework does this. It was the first tool to offer an application model for serverless architectures; now it’s the first tool to offer a uniform experience for deploying and managing serverless functions on every serverless compute vendor.
But we still need serverless standards. Each vendor has a different design for things like function signatures, which makes adoption and usability more challenging than it needs to be.
The team behind the Serverless Framework is working with the Cloud Native Computing Foundation and major cloud providers to converge on standards for serverless concepts. Discussions are still in their early phases, but all stakeholders have been extraordinarily collaborative, including larger vendors.
A Story of Functions …and Events
The serverless story has largely been focused on functions thus far. Yet, this is only the first half.
The other half of the story is about data — and since serverless architectures are essentially based on event-driven computing, their data is expressed in the form of events.
Now that serverless functions have enabled us to easily react to anything, everything is starting to look like an event: business events, application state changes, synchronous requests, asynchronous requests, notifications, messages, audit trails, system logs, errors, metrics, alerts, webhooks, clickstreams, health checks… to the serverless function, the world is filled with events, just waiting for action and analysis.
Events make data portable and liquid. They are the simple contracts serverless functions use to coordinate, regardless of where functions are located. They protect against vendor lock-in while enabling vendor choice. If there exists a single pattern that will bring order to the vast amounts of logic organizations will create over the coming years, it will be the event-driven pattern.
Unfortunately, event-driven tools and services currently exist in a haphazard state. In the future, these tools and best practices must evolve to sustain the levels of productivity we are now able to achieve.
Recently, the team behind the Serverless Framework announced a new type of open-source infrastructure titled the Event Gateway. The technology blends API Gateways and the Pub/Sub pattern to create a highly productive event router for serverless computing.
The Event Gateway is the missing piece of middleware for the serverless era, and can potentially become the backbone of the modern digital business. Its first goal is to express all data in the form of events (even raw HTTP requests). Its second goal is to route those events to any serverless compute provider, anywhere.
Event Gateway will enable teams to easily write serverless functions across cloud providers (or on-premise) and act upon them.
This project is soon to be followed by the Serverless Platform, which enables team collaboration across serverless applications and provides solutions for event data management. All of these tools will follow the serverless standards described above.
Total Automation and Intelligence
Automation is predicted to increase substantially — and necessarily so, given the world’s growing complexity and cost. To thrive in hyper-competitive and expensive times, every organization, team and sole individual needs the power of automation.
Along with the rise in automation will be the rise in systems of intelligence. An architecture that allows real-time, contextual data processing will allow businesses to make better decisions in the exact moment a problem arises.
Serverless computing and the serverless architecture can enable these scenarios, and their potential to do so cannot be understated — especially when you combine the ease of serverless functions with the scalable, organizational capacity of event-driven design. Functions and events are the two simple concepts we’ve been waiting for, to build vast systems of automation and intelligence, while keeping operational overhead minimized.
This is only the beginning of the serverless architecture and the story of serverless overall. While the potential is endless, to realize that potential we’ll need new types of tools and infrastructure to build and operate these next-generation serverless architectures.
This is what we are building at Serverless Inc.
The Cloud Native Computing Foundation, Google and Microsoft are sponsors of The New Stack.
Feature image via Pixabay.
