How to analyze memory leaks caused by EFCore

The terms "memory leak" and ". NET application" are not often used together. However, we recently experienced a series of out-of-memory exceptions in a. NET Core Web application. It turned out that this problem was caused by behavioral changes in Entity Framework Core, and although the final solution was very simple, the process of achieving this goal was both challenging and interesting.

该系统本身托管在 Azure 中，由 Angular SPA 前端和后端的.NET Core API 组成，使用 Entity Framework Core 与 Azure SQL 数据库进行通信。作为专门从事.NET 开发的软件咨询公司，我们之前已经编写了许多类似的应用程序。因此，内存不足崩溃是无法预料的，因此我们立即知道这是需要认真对待的事情。使用 Azure 门户中的指标，我们可以看到内存使用率稳步上升，然后突然下降：此下降是应用程序崩溃。

So we spent some time investigating and gradually making changes to resolve the seemingly classic memory leak problem. A common cause of NET leaks is the improper handling of certain issues, which in our case is likely the EF Core database context. Therefore, we walked through the source code to find potential reasons why the context might not be able to be handled. This became blank.

We have upgraded Entity Framework Core to the latest version because recent updates include fixes for various memory leaks and overall efficiency improvements.

We also found possible memory leaks in the version of Application Insights we were using (see https://github.com/microsoft/ApplicationInsights-dotnet/issues/594), so we also upgraded the package.

None of this solved the problem, so we dissected the memory dump we got from Azure application services (see https://blogs.msdn.microsoft.com/jpsanders/2017/02/02/how-to-get-a-full-memory-dump-in-azure-app-services/).

We noticed that the vast majority of managed memory is ultimately used by the MemoryCache class. Further in-depth research revealed that most of the cached data is in the form of raw SQL queries. We see a large number of events that are essentially the same query being cached multiple times, and the parameters themselves being hard-coded into the query rather than parameterized.

For example, instead of caching queries like this:

SELECT TOP (1) UserId, FirstName, LastName, EmailAddress
FROM Users
WHERE UserId = @param_1

We found multiple queries like this:

SELECT TOP (1) UserId, FirstName, LastName, EmailAddress
FROM Users
WHERE UserId = 5

So we did some searching for EF core issues that might be related to it and came across this issue: https://github.com/aspnet/EntityFrameworkCore/issues/10535.

The topic on this issue points out the problem: We are building a dynamic expression tree and using it Expressions.Expression.Constant to provide parameters to the where clause. Using constant expressions means that Entity Framework Core does not parameterize SQL queries and is a behavior change for Entity Framework 6.

We use this expression tree everywhere to get something through its ID, which is why it's such a big problem.

So here are the changes we made:

// Before
var param = Expressions.Expression.Parameter(typeof(T));
Expression = Expressions.Expression.Lambda<Func<T, bool>>(
  Expressions.Expression.Call(
    Expressions.Expression.Constant(valuesToFilter),
    "Contains",
    Type.EmptyTypes,
    Expressions.Expression.Property(param, propertyName)),
  param);
// After
var param = Expressions.Expression.Parameter(typeof(T));
// This is what we added
Expression<Func<List<int>>> valuesToFilterLambda = () => valuesToFilter;
Expression = Expressions.Expression.Lambda<Func<T, bool>>(
  Expressions.Expression.Call(
    valuesToFilterLambda.Body,
    "Contains",
    Type.EmptyTypes,
    Expressions.Expression.Property(param, propertyName)),
  param);

使用 lambda 表达式获取表达式主体会使Entity Framework Core对 SQL 查询进行参数化，因此仅缓存它的一个实例。

This is memory usage over time, including revisions. This version is marked in red, and you can see that there are significant differences. Stable memory usage never exceeded 200MB, but kept climbing to more than 1GB, and then a crash occurred.

When we initially investigated, the real solution was not something we had to pay attention to, but by checking the memory dump and following the evidence we finally got there.

The lessons that can be learned from this investigation are:

• Memory dumps don't lie-if memory leaks, check for evidence first.
• Microsoft has opened up the source code of EF Core, where all issues can be seen by everyone, making it very convenient for developers who need it.
• A simple code change (one line in this case) can have a huge impact.