Optimizing Entity Framework: Part 1

I was recently involved in a project where I used Entity Framework 4.3 (Code-First) for the first time. I must say I’ve enjoyed it more than I expected, it seemed to fit nicely with Domain Driven Design (DDD) principles we were following, migrations were quiet convenient to keep our incremental updates under control (most of the time) but most importantly the models were very focused and clean and as such, it didn’t feel like we were getting out of our way or bending our architecture/design to fulfil any constraints imposed by EF.

In this project – as it should be in most cases – we didn’t optimize prematurely. But towards the end of the project, with real data being used, it was obvious that EF was a bottleneck to our application and becoming noticeably slower than previous versions of the application (that used a combination of ADO.NET with MSSQL Stored Procedures and Linq-to-Sql). And this is when I began this interesting exercise of optimizing the performance of the application that I am sharing here.

Unfortunately, I can’t share the specific code as I don’t own it. But the application is a public facing website that uses ASP.NET MVC for its architecure and DDD complements the picture as mentioned previously. So at the front-end side of things, there are typically Controllers, highly ajaxified Views (making use of knockout and some other cool stuff that I will hopefully discuss in other posts) and Models corresponding to the Code-first models. Controllers talk to Services that talk to Repositories through their interfaces which in turn talk to the database using EF as the ORM of choice.

Tools of the trade: Profiling

In my view, if you’re talking about performance and not putting numbers on the table, then it’s just a chatter with no substance. With performance, speak numbers or remain silent.

So in this specific case, the aim was to decrease the response time from around 800ms-950ms to the range of 100ms. And this is the main reason why numbers are so important, because in many scenarios, a response time of 1s might be acceptable, for our case, it just wasn’t good enough especially when compared to the previous version of the application. I am sure that if you are developing for the scale of Facebook or Twitter then even 100ms could easily become as unacceptable. So measure it and decide.

In order to get started with the measurement exercise, you probably have one of two choices:

  1. Scatter around Stopwatch instances around your code and keep track of them.
  2. or you can use a Profiler.

dotTrace - Easily drill into the code stack and define your code bottlenecksI used dotTrace from JetBrains as my profiler of choice, it is not free (nor cheap) but it provides 10 days trial which was more than enough for this exercise. Most of the profilers – code execution and performance profilers rather than memory profilers – out there are commercial and expensive (for personal use at least), the only free option seems to be EQATEC which I only had a quick look at but it seems like it could do the trick as well.

So running dotTrace, I could see exactly where the bottlenecks in my code where. They were indeed around Database access (don’t take that for granted) and Entity Framework.

Tracing EF generated Sql

EF can surprise you when it comes to generated SQL, and this is probably where most of the optimization will happen. So you need an easy way to check what is being generated by EF, at the end of the day, whatever you’re writing in C# and linq, no matter how fancy, testable, well written it is, it gets translated to the only language your database understands: SQL.

So here is the other amazing tool that you need to add to your toolset: Entity Framework Profiler. As with most of these tools, it is not free nor cheap, but you get a 30-days free trial. If you can’t afford it, the other option would be to use SQL Profiler to trace the SQL generated by EF (it is also not free but your organization could probably have it as part of their SQL Server edition anyhow). Worst case scenario, you could go back to basics and use .ToTraceString() to have a peak on the generated SQL.

Entity Framework Profiler: Easy insight into what EF is generating on your behalf

Now, you probably could survive without both of these tools, by peaking into your code through debugger, Stopwatch, trace messages or any other means. But since performance tuning is a highly iterative exercise where you need to repeat and compare many times before reaching a satisfactory conclusion, then having such tools is highly rewarding. It frees you from one extra point of pain and let you focus on the task at hand.

So I've decided to blog technical …

… something I’ve always wanted to do, but never felt confident enough to start. Now, I decided to just do it. First, I believe that everyone has some knowledge/experience worth sharing. Second and most importantly, as someone once put it: “to teach is to learn” so by sharing and teaching the things I know about, I will hopefully learn something.

So my blog is now divided into two sections:

1. Rai* where I will continue to express my random thoughts about random issues often about Sudan and being Sudanese. Topics in this category will continue to be rough, irrational and angry at times but so are the topics they cover.

2. Teknologia for my techy ramblings. My aim – for a start – is to write consistently regardless of the quality and am sure that the quality of my entries will improve if I keep the habit of blogging. A “relevant” post every two weeks is my initial target.

Language remains a problem and I am undecided whether to write in Arabic or English. For technical subjects, I prefer English as it is – whether we like it or not – the language of technology, and non-English speakers (Arabic or otherwise) have to make the effort to learn it, at least in the context of IT, otherwise they will be left behind. As for the “Rai” pages, they will continue to be a mix of both depending on the topic, audience and mood.

* Raï (English pronunciation: /ˈraɪ/; Arabic: راي‎) is a form of folk music that originated in Oran, Algeria from Bedouin shepherds, mixed with Spanish, French, African and Arabic musical forms, which dates back to the 1930s.
Singers of raï are called cheb (shabab, young) as opposed to sheikh (shaykh, old), the name given to Chaabi singers. The tradition arose in cities like Oran, Relizane and Mostaganem, primarily among the poor. Traditionally sung by men, at the turn of the 20th century, female singers became common. The lyrics of raï have concerned social issues, such as disease and the policing of European colonies, that affected native populations. (Excerpt from wikipedia)

خاطرة موامبا

وهناك تلك المرة التي ظننت فيها أنني متّ.

لم أر نورا في آخر الممر، لم يكن بجانبي ملائكة ولم تمر لحظات حياتي أمام عيني. كنت نائما نوما عميقا، نوم أعرف حق اليقين أنه الموت ووددت لو أصحو للحظة لأخبر كل الخائفين من الموت أنه مجرد غفوة.

لم أدرك الحكمة ولم أصل للنيرفانا ولم تكن لي روح. فقط جسدي وأحلامي كما كل ليلة. ولم أتذكر ابني أو صديقتي، فقط أحلام متفرقة بعضها مزعج وبعضها أقل إزعاجا كأي غفوة.

لم أتذكر لحظاتي الأخيرة ولم أفكر في المباراة ولم أتمن أن أكون في مكان آخر أو مع أشخاص آخرين ولم أندم على شيء. لم أحزن لترك حبيباتي ولم أنتظر قبلة وداع.

لم أندم. ولم أندم على أنني ندمت يوما.

لم أهتم بدموع من أحبوني. أدركت أن كلهم لهم أحلام بعضها مزعج وبعضها أقل إزعاجا، وأنهم سينامون ولن يندموا ولن يهتموا بدموع من أحبوهم.

كنت في حالة صفاء، صفاء أن توقن وأن تؤمن. صفاء لا يهم ما قبله ولا يوجد ما بعده.

وفجأة استيقظت.

وعرفت أنها لم تكن سوى مباراة أخرى غير ودية مع القدر. لكني لم أهتم.

** من وحي فابريس موامبا الذي توفي لدقيقتين ثم عاد

Sudan: When you corner a dictator!

For the brave students of Sudan, the 30th of January was not a normal Sunday. It was a day that could enter history. Inspired by the Tunisian and Egyptian revolution, thousands of Sudanese students dreamt of their own version of a Facebook revolution.

For Sudan though, the story has its unique complications with a regime that managed to take dictatorships into a new low. The regime, which is a military dictatorship that has overthrown democracy since 1989 , has a long history of misbehavings: The civil war with the south and the Darfur conflict are much publicized in the Western media but some of the less publicized crimes include dragging thousands of teenagers into the frontlines of a civil war in the nineties, hosting Bin Laden and his Jihadis for several years in mid-nineties, and last but not least the so-called Ghosts Houses where thousands of students, professors and activists were regularly tortured and killed. The head of the regime – Omer Albasheer – is being prosecuted by the International Criminal Court for crimes against humanity, he’s the first head of state to be prosecuted while in power. For the president and his regime, sticking to power is inevitable. The brave Sudanese students are faced by one of the worst kind of dictatorships: a cornered one.

On January 30, the first attempt for protests was brutally confronted by security forces. A student was killed by security men wearing civil clothes, he’s not the first student to be killed during this 22 years dictatorship nor the last I am afraid. Hundreds were detained for days, some of them still missing and reported to be under continuous life-threatening torture. Several attempts to gather and reorganize the lines of protesters has been made, but the major obstacle was that the fight – admittedly – hasn’t caught much momentum outside of universities.

The turning point – though – might prove to be a young brave Sudanese girl called Safia Ishag. She is a fine arts student who was raped by 3 officers from the criminal National Security Bureau for her role in organizing the 30th of January protests. She bravely decided to confront the regime with its crime in an emotional video. Safia’s bravery managed to unite people behind her, for the first time, the students’ calls started to make it outside universities’ walls. Several groups decided to take on the streets starting the 4th of March and they are seeking new ways to organize their lines against the density and brutality of security forces.

So how can we help? Believe it or not, but a Facebook share or a Tweet could be enough. The regime – like any other dictatorship – thoroughly targets the youth’s confidence, they force media blackout in order to stop people from gaining the momentum, they’ve learned from other dictatorships’ mistakes. In North Africa’s revolutions, social media played a crucial role in giving the revolutionaries the confidence they need in their battle. Knowing that someone out there listens and sympathizes with your cause can have a magical effect.

So Tweet, Share and Tell your colleagues, friends and family about Safia and her brave friends. It’s the least we could do to honor those brave warriors.

ثورة سالي واخواتها

فكرة الثورة هي ثورة حالمة بكل المقاييس، فسواء كانت الفكرة مقرونة لديك بنضال مارتن لوثر السلمي أو جهاد صلاح الدين الأيوبي أو جيفارا أو حتى الهيبيز في أوربا وأمريكا الستينات فإن ما لا شك فيه أننا جميعا – بغض النظر عن الأعمار والجنسيات والثقافات – كلنا نتوق لثورة ما. على أرض الواقع، فإن هذه الفكرة الحالمة لديها قابلية كبيرة أن تتحول لكابوس. ربما لا توجد ثورة في التاريخ لم يتم سرقتها من الساسة أو رجال الدين والجيش المنتفعين.

لكن ثورة مصر مختلفة. هذه الثورة ليست ثورة جياع كما حدث ويحدث في بلدان كثيرة وفي عصور كثيرة، وهي ليست ثورة سياسيين مهما حاول البرادعي أو الاخوان المسلمون أو غيرهم ادعاء ذلك. هي ثورة الشهيدة سالي واخوتها واخوانها. وما يميزها حقا، أنها ثورة لم تحدث فجأة أو في لحظة غضب، فهي ثورة مع سبق الإصرار والترصد.

مصر – وجيلها الجديد – قرروا تغيير النظام وفعلوها. سأشدد على أنها ثورة الجيل الجديد ﻷن النظام الكهل الذي تمرس في قهر المصريين بالطرق القديمة، هذا النظام الكهل فوجئ بجيل جديد وأساليب جديدة لا مركزية لا يمكن السيطرة عليها. شباب مصر العظيم قرروا ونفذوا وهذا ما لم يحدث في التاريخ. شباب مصر يسطرون التاريخ الآن وكلي يقين – أنه على خلاف ثورات الماضي – فإن هذه الثورة لن تسرق لو حماها الشباب حتى النهاية. وهنا مربط الفرس، حتى النهاية وهذه النهاية ليست فقط إسقاط مبارك بل ما بعد إسقاط مبارك.
Continue reading “ثورة سالي واخواتها”

Random thoughts about Egypt

In Egypt, you always hear a variation of this phrase: “Egypt is bigger than anything”, it doesn’t matter whether the context is as serious as regional politics or a mere football game. Egyptians truly believe and think that Egypt is bigger than anything. And this is the soul of what makes Egypt, Egypt.

Egypt – in my view – is a land of striking contrasts. Egypt, along with Ethiopia, are probably the only two countries in Africa that existed before the British and the French decided so. It is not just political borders drawn by colonialists as it is the case of most other countries. On the other hand, Egypt has not been ruled by an Egyptian for more than 3 millennium until Mohamed Naguib took power.

Egypt was the center of all political and social movement in the middle east in the past century. When Egypt embraced liberalism in the early century, the whole middle east went liberal. When it went anti-colonialist, the whole region followed. When it embraced pan-Arabism, the whole middle east was after Nassir. And when it went fundamentalist, the whole middle east went fundamentalist as well. Like it or not, it all starts in Egypt, for good or for worse. This is in contrast with an apparently very stagnant political life that – in recent history – has no mention of any revolutions, no coup d’etat and a huge number of divine leaders who ruled until death.
Continue reading “Random thoughts about Egypt”

Hierarchy of human needs: Food -> water -> Passport

I must say that my obsession with getting a different passport is hardly unique to me. I could argue that it is actually the one thing that unite the majority of young Sudanese (if not African) of today. It is bitterly funny (and unlikely to be a coincidence) that decades after independence, the new generation of Africans is eagerly trying to get rid of the identity that our fathers fought for.

My personal obsession started as a Sudanese boy in Egypt. The ultimate dream for most of us – as teenagers when you really start feeling the burden of your inherited identity – the ultimate dream was getting an Australian, Canadian or American passport (these were the main alternatives). We all had a story of a “Sudanese” who immigrated to one of those countries and when coming back to Egypt/Sudan/wherever, being harassed by a policeman, he “courageously” shows his Carte Blanche to misbehave in the third world: his newly acquired, hardly fought for, digitally signed, Holy Passport.

Being in Europe and having the possibility/mood for looking back at things and thoroughly analyzing them, I remembered my Economics teacher in school. He used to love to explain Maslow’s hierarchy of human needs, he liked to emphasize how we – the lucky ones who made it that far at school – how we lie at the bottom of that pyramid (since we were even deprived of the basic need of sex). Regardless of the sexual dilemma in these countries, our teacher always emphasized the value of respect of others to go beyond the first level of the pyramid and how it is what makes fulfilling the rest of the needs a possibility. He couldn’t be more correct.

Lack of respect of others is at the heart of it. Whether it is the ugly policemen of Khartoum who give themselves a divine right to beat the hell out of a woman because of wearing trousers, or people thinking that they are higher than the rest because of tribe or religion, or simply a teacher at school beating kids to fulfill his own lack of respect to himself. Lack of respect – on a higher level – is what made Apartheid, it is what made genocide, it is the cause for the rise of the likes of Bush and Bin Laden, it is what made Hitler and what will make the next Hitler.

This lack of respect creates anger. And it only takes a businessman and/or a politician to transform that anger into something more devastating.

My teacher never proposed solutions to fight this lack of respect, he was not the solution-proposing type of dude. Others – being more idealistic – proposed revolution, the majority clung to Religion as the solution. I was not as courageous as the first group or as naive as the second one, my solution was more pragmatic, it goes somewhere within the line of “Get a fucking passport, climb that bloody pyramid and show your passport in the face of that ugly policeman”.

But regardless of the means, all of us share the same goal: it is absolutely vital for us to climb that pyramid and it is absolutely vital for us to gain the respect of others (and eventually self-respect). It is a need as crucial as food, water … and sex I assume.

to be continued …

Maslow's Hierarchy of Human Needs
Maslow's Hierarchy of Human Needs