How to Analyze Memory Leaks Caused by EF Core

The terms "memory leak" and ".NET application" are not often used together. However, we recently encountered a series of out-of-memory exceptions in a .NET Core web application. It turned out the issue was caused by a behavioral change in Entity Framework Core, and although the final solution was very simple, the process of reaching it was both challenging and interesting.

The system itself is hosted in Azure, consisting of an Angular SPA frontend and a .NET Core API backend, using Entity Framework Core to communicate with an Azure SQL database. As a software consulting company specializing in .NET development, we've written many similar applications before. Therefore, out-of-memory crashes were unexpected, and we immediately knew this required serious attention. Using the metrics in the Azure portal, we could see memory usage steadily rising and then suddenly dropping: that drop was the application crashing.

So we spent some time investigating and made incremental changes to address what seemed like a classic memory leak. A common cause of .NET leaks is something not being properly disposed, in our case likely the EF Core database context. Therefore, we went through the source code looking for potential reasons why the context might not be disposed. This turned up empty.

We upgraded Entity Framework Core to the latest version, as recent updates included various memory leak fixes and overall efficiency improvements.

We also discovered a potential memory leak in the version of Application Insights we were using (see https://github.com/microsoft/ApplicationInsights-dotnet/issues/594), so we upgraded that package as well.

None of these solved the problem, so we dissected memory dumps taken from the Azure App Service (see https://blogs.msdn.microsoft.com/jpsanders/2017/02/02/how-to-get-a-full-memory-dump-in-azure-app-services/).

We noticed that the vast majority of managed memory was ultimately occupied by the MemoryCache class. Further investigation showed that most cached data was in the form of raw SQL queries. We saw a large number of events where essentially the same query was cached multiple times, and the parameters themselves were hardcoded into the query rather than being parameterized.

For example, instead of caching a query like this:

SELECT TOP (1) UserId, FirstName, LastName, EmailAddress
FROM Users
WHERE UserId = @param_1

We found multiple queries like this:

SELECT TOP (1) UserId, FirstName, LastName, EmailAddress
FROM Users
WHERE UserId = 5

So we searched for related EF Core issues and came across this one: https://github.com/aspnet/EntityFrameworkCore/issues/10535.

The issue described the problem: we were building a dynamic expression tree and using Expressions.Expression.Constant to supply parameters for the where clause. Using constant expressions means Entity Framework Core does not parameterize the SQL query, and this was a behavioral change from Entity Framework 6.

We used this expression tree everywhere to retrieve something by its ID, which is why it was such a big problem.

So here's the change we made:

// Before
var param = Expressions.Expression.Parameter(typeof(T));
Expression = Expressions.Expression.Lambda<Func<T, bool>>(
  Expressions.Expression.Call(
    Expressions.Expression.Constant(valuesToFilter),
    "Contains",
    Type.EmptyTypes,
    Expressions.Expression.Property(param, propertyName)),
  param);
// After
var param = Expressions.Expression.Parameter(typeof(T));
// This is what we added
Expression<Func<List<int>>> valuesToFilterLambda = () => valuesToFilter;
Expression = Expressions.Expression.Lambda<Func<T, bool>>(
  Expressions.Expression.Call(
    valuesToFilterLambda.Body,
    "Contains",
    Type.EmptyTypes,
    Expressions.Expression.Property(param, propertyName)),
  param);

Using a lambda expression to get the expression body makes Entity Framework Core parameterize the SQL query, so only one instance is cached.

Here is the memory usage over a period of time including the revision. The release is marked in red, and you can see the big difference. Stable memory usage never exceeded 200MB, whereas before it climbed to over 1GB and then crashed.

When we initially investigated, the real solution was not something we noticed, but by examining memory dumps and following the evidence we eventually arrived there.

The lessons learned from this investigation are:

• Memory dumps don't lie - if there's a memory leak, look at the evidence first.
• Microsoft has open-sourced EF Core, and all issues are publicly visible, which is very convenient for developers in need.
• Simple code changes (in this case one line) can have a huge impact.