Agile Big Data: How I Learned to Stop Worrying and Evolve with On-Demand Schemas
The web app is Agile. The REST API is Agile. The testing and planning are Agile. But alas, Data infrastructures certainly are not. Once an application matures, changing the shape or indexing scheme of data often forces at best a top-down planning exercise and at worst includes schema changes which force downtime.
This talk will discuss solutions to some of the problems faced by software organizations as their products become successful and the data grows beyond the initial scope of the database, whether it starts out as relational SQL, NoSQL, or somewhere in-between.
As the user base of a software system grows, the data infrastructure is forced to evolve to meet the needs of new features, increasing the variety of data with ever larger data volumes and higher query velocity. One thing we know about evolution is that it doesn't always take the simple path or produce simple results. Data systems often grow by adding technology: a little NoSQL here, a message queue there, a Hadoop cluster hiding in the corner. This growing complexity slows down product development and challenges technology teams to ramp up entirely new skillsets reactively.
Is this your future or is this your today? Address these challenges, with a truly agile infrastructure that allows your organization to store, scale and access the data it needs.
We will cover the concept of an 'Agile data infrastructure', able to handle everything from OLTP to real-time analytics to historical reporting. A truly Agile approach allows data to be accessed as an exact match to application requirements. As application needs change, an Agile platform must be able to respond rapidly and dynamically evolve the schema, data distribution and access patterns to support high scalability with high performance - while easing the burden of the application developer and database administrators.
We will also highlight real a real-world use case from a large enterprise organization that was able to leverage an agile data approach to rapidly accelerate the pace of product development while actually simplifying operations.
We'll make you a believer in the fundamental tenants of an agile big data infrastructure, namely that it must be Simple, it must be Scaleable, it must be Streaming, and, most importantly, it must be SQL.
Dan Lynn, CEO of AgilData, formerly CodeFutures Corporation, is an entrepreneur, technologist, teacher and technical leader with 14 years of experience in the software industry. He joins AgilData to advance the agile Big Data strategies in development around AgilData, CodeFutures' new agile Big Data platform designed to remove the complexity and limitations of current data management approaches.
Dan previously was the founding CTO at FullContact Inc., a contact management startup focusing on fuzzy match and record linkage problems. Dan is an accomplished speaker and is actively involved in the Colorado software community, serving as an organizer for the Denver Storm users group and the Boulder/Denver Big Data meet-up group. In addition, he is also a Techstars alumnus - Boulder 2011, with FullContact.
Cory Isaacson is CTO of Risk Management Solutions. As CTO, he drives the delivery of the advanced, cloud-based RMS(one) platform, leading the industry with innovative approaches to software management and advanced technology for the complex and unique challenges of risk modeling. Prior to joining RMS, Cory was CEO/CTO of CodeFutures Corporation, maker of AgilData and dbShards. Cory has founded or held executive positions in several software-related businesses, and authored the books Software Pipelines and SOA and Understanding Big Data Scalability.
He has more than 20 years of experience with advanced software architectures, and has worked with many of the world's brightest innovators in the field of high-performance computing. Cory has authored numerous articles and spoken at hundreds of public events, helping organizations address the real-world challenges of Big Data performance and scalability, and effectively applying emerging technologies to fields including social networking, mobile applications, gaming, and complex computing systems. He holds a BA from the University of California, Santa Barbara.