SQL Horror

Recently I came across a comparison of Transactional DDL support in various databases, on the postgresql wiki.

For me  this has been an eye-opener – having started my career with Oracle databases, I was brought up thinking that ddl statements are always outside transactions, without questioning why it must be so.

PostgreSQL not only supports transactional DDL, it does so in the most intuitive way – you just wrap these statements in a transaction block and everything can be rolled back if necessary. No auto-commits (like Oracle/MySQL does), no need for specific Isolation levels (like SQL Server) – works quite robustly without any problems.

Just one more reason for me to pick Postgres as the database of choice for my upcoming projects. Then of course, there is that thing about Postgres actually being the most advanced database and open source at the same time.

Now that I am learning Kannada from scratch, I started searching for effective ways of doing so – here are some of the things I have found useful so far.

Kannada Baruthe – Useful starter website – collection of daily usage words and phrases, along with audio – great for beginners. The content is broken into several categories such as Starters, Greetings, Enquiry, Directions, Relationships, Numbers and more, so that it is easy to learn and get used to the context.

A couple of useful Mobile apps – these are great because you can use them when travelling, when you are generally doing nothing else.

Kannada Kali – A basic app, decent enough to get you started with some basic conversations. There is a cool feature to practice writing basic alphabets, although it would have been nice if they had given the english phonetics of the alphabets in the same place – that’s in a different category altogether (so you’re expected to know the alphabet when you actually practice writing them)

Learn Kannada – this one’s good because it allows you to search through it’s content so it’s easy to find something – extremely handy, especially for quick reference. However it does crash a lot, I think my poor Samsung Galaxy pop can’t handle this search all that well

Of course, nothing beats google translate when it comes to ad-hoc translations on either sides.

We had a question today on what’s the real advantage of going with REST over SOAP. It’s tempting to say that REST is the next best thing after sliced bread while SOAP is the devil’s spawn but here’s a more reasonable attempt at explaining the difference –

• SOAP is a protocol on top of HTTP whereas REST is a design principle of designing services with basic HTTP protocol.
• REST uses HTTP verbs GET, POST, PUT, DELETE, etc for the things that they were intended – SOAP is mainly an RPC with XML that primarily uses POST requests
• REST doesn’t try to handle state; SOAP protocol is designed to handle state of a particular client through session (which adds a bit of overhead)
• HTTP is already handled by browsers and all web clients, as well as any web server – on the other hand SOAP protocol needs an additional layer of software, and each vendor has it’s own implementation (which may or may not fully adhere to the SOAP protocol).
• SOAP exposes API in form of WSDL which allows client-side code generation – this makes it easier to get started but also introduces maintainance issues
• Versioning is more difficult with SOAP – even additional properties getting added will break the clients if they are on a previous version – on the other hand, versioning is much simpler with RESTful services, since any additional properties added on the server side are just ignored at the client side if they are not required
• If there are strict contracts to be forced between a service and a Client, SOAP has explicit ways of doing that – REST has no such mechanisms
• A lot of client side libraries (such as backbone.js) make it much easier to consume RESTful web services
• Since SOAP is a huge abstraction on top of the HTTP protocol, things get really difficult to debug when the abstraction breaks down – for e.g. when a configuration has a problem. Debugging issues in HTTP calls is relatively much easier since there is no proprietary stack in between your code and the client calls.

Can you think of any other differences?

UPDATE: I recently posted an interview with Demis Bellot on InfoQ that explains in a lot more detail the perils of SOAP – you should read it.

Google recently published a paper describing Spanner, their globally distributed, Paxos-replicated database with externally consistent transactions, using specially designed hardware and a new TrueTime API.

The motivation behind using Spanner for certain types of applications instead of the massive key-value store that is BigTable is interesting –

We believe it is better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise, rather than always coding around the lack of transactions.

In an effort to achieve this, Spanner combines the Replicated ACID transactions of Megastore with the Scalability and throughput of BigTable. The challenges faced and the solutions around this (especially custom hardware and the TrueTime API) are described in great detail in the research paper. You can also view the presentation “Building Spanner” from Alex Lloyd (or just download the slides) to get more details.

Google has already moved F1, the backend for their advertising platform, to use Spanner instead of MySQL.

There are several reactions from around the web.

nlavezzo of FoundationDB observes

It’s interesting to see that the creators of BigTable and the early proponents of eventual consistency have invested the last 4.5 years building a system that adds back strong consistency guarantees.

If the Spanner paper is as important as BigTable, ACID may become the new goal for those building distributed systems.

linuxhansl who works on Apache HBase remarks

It makes me sad to see how far ahead Google is compared to the rest of the world. The notion of uncertain time is ingenious.

Mike Miller of Cloudant Labs says –

It represents a pragmatic acceptance of developer’s reluctance to reason in the absence of immediately consistent transactions and therefore strikes a bittersweet chord. Philosophically this feels like a big step forwards for distributed systems.

It is probably too early to see whether open source software similar to this will appear soon, especially given that Google leverages custom hardware to really benefit from this. Also we don’t know whether Google will offer this as an external data service through it’s Cloud platform. Such software though could definitely help companies leverage globally distributed cloud infrastructure (to avoid service disruptions when a data-center goes down) and still rely on strong consistency guarantees when designing their applications. NuoDB and FoundationDB are at least two projects that seem to target such a use case although it is unclear how they will compare without special, time-handling hardware.

I’ve been spending some time with entry-level and mid-level .NET programmers who have gone through several courses related to C#, WCF, ASP.NET, Windows Forms, and various technologies. However most of them don’t know the first thing about unit testing, Design patterns such as dependency injection, practices such as continuous integration, Acceptance and integration test automation and so on. If you think about it, technologies change, but practices don’t and it’s high time we start focussing our courses on practices rather than the technology that’s currently in flavor.

So I’ve kept an aim to come up with some easy to digest courses that will hopefully encourage and enable average programmers already well-versed with C# to push forward and start using some of these useful practices. This is what i have in mind so far – all the below will contain code samples and hands-on exercises so that the learners can start applying them in their real-world projects

1. Unit Testing
2. Dependency Injection
3. Automated Integration and Acceptance Testing
4. Continuous Integration
5. ORMs

Of course these look basic and there are many libraries and tools that we can teach about, but I think some of these concepts are essential and go beyond language and framework concerns. Some hands-on is must to get them imbibed into the programmers’ toolkit.

What do you think? Any suggestions to improve this course outline?

For a few days now, I’ve been trying to wrap my head around how small startups in India are actually having any margins in the outsourcing development business. Challenges seem to be plenty, and a lot of people seem to be getting into it these days.

For instance -

• Sticker-price for developer hours seem to be in a downward spiral – it’s not hard to see either freelancers or small companies offering development resources at as little as $12/hr. Of course quality is generally suspect, but customers who blindly compare hourly rates couldn’t care less. What’s worse, is that the experience they have (with poor quality deliverables) is often extrapolated to the entire industry.
• Industry-wide salaries seem to be increasing – sometimes even unreasonably so. A side effect of everyone trying to start this business, is that there is intense competition for resources – so much so that a lot of average or under-average software engineers also tend to get paid a lot. Competition is intensified with even product-based, ecommerce or other VC-funded businesses coming into the picture. There is almost no correlation between the salaries for Software Engineers compared to other jobs requiring similar skill levels in other industries, which is a worrying trend.
• Increasing salaries is drawing almost everyone into the software industry – engineers are blindly getting into software companies irrespective of their majors – more a rule now than an exception. And a mediocre software developer is almost an expectation these days – it’s *almost* impossible to find fresh graduates who actually *love* developing software
• We are actually running out of resources – large companies have started hiring BSc graduates instead of Engineers for instance, and are lowering their filters to ensure that they are able to staff their work-force.
• Giant outsourcing successes such as Infosys, TCS, Wipro, Cognizant, IBM seem to be the ideals that a lot of entrepreneurs strive for – without realizing that these were started in a different age, and it is nigh impossible to replicate such a feat with the current market dynamics, especially as a startup

The Industry might seem great for an economy like India – lots of jobs created, good amount of exports generated, talent pool created within the country and so on – and indeed in many ways it has contributed to the bulging middle-class in the country. But the fact remains that a lot of Indian companies do not do any “high-end” technology work – or focus on finding a niche that they can truly do justice to. To top that off, high stress levels, 60 hour work weeks, high attrition levels, etc. seem to be just getting factored into the business plan – a very worrying sign indeed.

Now don’t get me wrong, there are some companies that understand the need to differentiate themselves, take care of their employees and ensure that they create unique value for their customers, but this is not true about the industry as a whole.

These problems are faced by large companies as well – for instance, Infosys is very stringent about the margin for any new business they take up, but it was recently overtaken by Cognizant in terms of revenues. Cognizant is known for their lesser margins, and still lag Infosys in terms of net-profit. So does this mean that the only way to keep growing here is through reducing margins?

What is the end-game to all this? Will the industry just implode onto itself when this becomes unsustainable? Or will it grow out of pure-outsourcing and focus on consulting, product-development, platforms and other high-margin services that offers more differentiation opportunities? And what about startups, where all these problems get compounded several times?

I think it’s time we stop thinking of software development as an arbitrage business and start focussing on providing value that rivals the best consulting/development firms in the world. Competing on price alone seems to be a strategy doomed to failure in the long run. And if you are a start-up trying to get into the services business, you better have a differentiator/niche that allows you to charge more than the market rates. Else your business is will either fail or may become a mediocre success at best.

Someone asked on a mailing list how do you find out who’s “startup material”.  Here’s my take -

• Stupid enough to start a company
• Naive enough to trust others (partners, customers, employees)
• Dumb enough to work hard for lesser pay, when you could be earning more with less work
• Insanely patient through n-number of failures
• Extremely optimistic under any circumstances
• A stress junkie
• Ability to involve other “startup material” folks in your idiotic plan
• All in hopes of a rich pay day and an opportunity to “do something” (yeah right, as if that’s important) even if the statistical probability of it actually happening is 0.01% (note the point about extreme optimism above)

So, are you “startup material”?