May 23rd, 2013 AppNexus: Engineering at Scale: The Ins & Outs of Shipping A Scalable Product
On Thursday, May 23, 2013, OLC attended Engineering at Scale: The Ins & Outs of Shipping A Scalable Product held at AppNexus. The event featured Theo Schlossnagle, the CEO of OmniTI and Circonus.
Theo Schlossnagle builds teams to solve today’s hardest scalability problems. Schlossnagle is an engineer, a practitioner of academic computing, an IEEE member and a senior ACM member.
Schlossnagle began with talking about data storage. “Everyone uses a different approach to storing data. The conundrum is to scale up or down. It’s a simple question, but people often arrive at the wrong solution. You need to plan your problem, project your possible audience over two to four years. You need to map the technology. And of course, keep in mind that new does mean dangerous. Now, that doesn’t mean don’t use it. Just be aware that it’s unproven,” Schlossnagle said.
He presented that people have a common misconception that SQL is the cure-all database. “Yes and no,” Schlossnagle said. “SQL is just a structured language for asking questions about data with relationships. You see, ACID can’t scale. The CAP theorem says so. But the CAP theorem is complicated—most people don’t understand it. It stands for Consistency, Availability and Partition-tolerance. There are many solutions, but the best way is to take your problems and partition them.”
Schlossnagle explained that noSQL systems vary widely. “Some are about NSPI [No Single Point of Failure]. Some are just key-value storage. Others leverage consistent hashing. Most sacrifice CAP. Again, people have common misconceptions about noSQL. They think it’s the silver bullet—well, there aren’t any silver bullets and noSQL isn’t the answer.”
“There are some problems where we can’t afford to store data. We want answers to our question. This leaves the realm of data stores and enters Complex Event Processing, Intelligent Event Processing and Streaming Processes. Databases are the problem that every gets stuck on. We underestimate the number of people that are interested in our data. Using a procedure-consumer paradigm for messaging, new consumers can be asked without knowledge of the publisher,” Schlossnagle said. “If you look at a CEP pattern, start with the IEP server. Use snippets of Java code then ask things to Log. All you need is a big enough pipeline to utilize this and make data available too. 
For his third point, Schlossnagle described caching. “This is how the internet works. 
There’s many levels of caching. It goes from the bottom to out-of-the-box caching. 
Different data sources will be different, calculating costs for a consistency and accuracy guarantee is factored into the plan.”
“Each type of data should have its own caching policy. You don’t have to pull all of this data by action—think lazy evacuation—but practically, not theoretically. 
He talked about two things that can be done to prevent this:
1) monitoring. A lot of software ships with metrics. A lot of the metrics are actually worthless. 
2) The problem in large-scale corporation is that work is siloed off frequently.
“Your business is likely not an infrastructure service. You don’t sell storage, IOPS, CPU cycles, but you’re selling consumer and business services [if you are in cloud]. The idea tells us that 10 percent connectivity is what powers your business—it probably is.
Next, Schlossnagle talked about monitoring what matters. “Correlate, project, plan and question,” Schlossnagle said. “Project your data, know your business model and understand its accuracy. This is capacity flowing,” he said.