My application runs just fine when compiled without EurekaLog.

However, if I compile it with EurekaLog:

• I am being stopped at the ELowLevel.DebugBreak being called from EBase.PanicModeOn when run under debugger;
• If I continue execution (or simply run outside debugger) - I get the following error dialog:

Reason:

EurekaLog has memory checks enabled by default. Please note that "leaks checks" and "memory checks" are two different things. You can enable memory checks while having leaks checks disabled (which is a default case). On the other hand, you can't have leaks checks enabled while memory checks are disabled.

If memory checks are enabled: each memory allocation in your application will have a larger size to store extra information (header/footer) required for checks to work:

• Leaks checks are disabled (default): each memory allocation will be larger on 32 bytes (to store block's info, flags, sentry values, see ETypes.TPointerDummyRecord, ETypes.BlockDummyOverHead);
• Leaks checks are enabled: each memory allocation will be larger on 272 bytes (to store call stack, see ETypes.TPointerRecord, ETypes.TPointerFooter, ETypes.BlockOverHead, ETypes.BlockOverFoot).

For example:

1. If you allocate 8 bytes 1'000 times: your application would allocate (8 + 32) x 1'000 = 40'000 bytes instead of the usual 8'000 bytes resulting in the overhead from EurekaLog (not including overhead from memory manager itself) of 32'000 bytes (x5 overhead);
2. On the other hand, if you allocate 200 bytes 1'000 times: your application would allocate (200 + 32) x 1'000 = 232'000 bytes instead of the usual 200'000 bytes resulting in the same overhead of 32'000 bytes, but now it is just x1.16 overhead;
3. Finally, if you disable memory checks altogether (the "Enable extended memory manager" option): EurekaLog won't be installing a filter on your memory manager, thus all your memory allocations will be unaffected by EuekaLog resulting in 0 overhead.

Conclusion: the larger your allocated memory blocks are = the less overhead. Worst case is like allocating simple TObjects - which are just 8 bytes (.InstanceSize method).

Diagnostic:

• The "Allocating" value indicates argument for the GetMem, AllocMem or a similar function. It indicates how much bytes your application tried to allocate when it failed. This value may be missing - for example, if memory checks are disabled (so RTL's default memory manager is used).
• The "Largest available" value indicates the size of the largest continuous free (unoccupied) memory block in your application. This value is obtained by scanning your address space looking for a MEM_FREE region of a largest RegionSize.
• The "Total allocated (RTL)" value indicate how much memory was allocated via RTL memory functions (GetMem, AllocMem, etc.). It can be either a single value or two values: 
  • One value: it is shown when EurekaLog memory filter is not installed, so all allocations go through the default RTL memory manager (or a 3rd party memory manager, if you have one installed);
  • Two values: it is shown when EurekaLog memory filter is installed, so allocations go through EurekaLog. Two values indicate total size of all allocations without and with overhead from EurekaLog.
• The "Total allocated (virtual)" value is equal to ullTotalVirtual - ullAvailVirtual from the GlobalMemoryStatusEx. It indicates how much memory is allocated (or reserved) in your address space. This includes loaded modules (exe, DLL, bpl), memory allocated via RTL, memory allocated by 3rd party code (system, DLLs, etc).
• The "Total available (RAM)" and "Total available (page file)" indicate free space in RAM and page file respectively.
• The "Total (RAM)" and "Total (page file)" indicate total size of RAM and page file respectively.

You need to analyze these values in order to understand why "out of memory" exception was thrown.

IMPORTANT: keep in mind - the memory is not static, it is always changing. It means, the memory will be changed between the moment exception was thrown and bug report was created. Therefore, various values from the error message and from the bug report may not be 100% accurate. Don't expect a perfect match, but look for generic patterns - see examples below.

A). If the "total allocated (virtual)" value is close to the possible maximum of address space for your application:

• 2 Gb for normal 32-bit app;
• 3 Gb for large address awared 32-bit app on 32-bit system;
• 4 Gb for large address awared 32-bit app on 64-bit system;
• 8 or 128 Tb for 64-bit app (depends on Windows version).

then you have a real out of memory error, because you have exhausted all free memory in your application.

For example:

Allocating: 1112 (1.09 Kb)
Largest available: 1183744 (1.13 Mb)
Total allocated (RTL): 1887792563 (1.76 Gb) (1945427109 (1.81 Gb))
Total allocated (virtual): 2076430336 (1.93 Gb)
Total available (RAM): 21130706944 (19.68 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (RAM): 34297835520 (31.94 Gb)
Total (page file): 38592802816 (35.94 Gb)

As you can see, the "total allocated (virtual)" value is 1.93 Gb, which is pretty close to the possible maximum of 2.0 Gb for a normal 32-bit app. Therefore, it is an error message about real out of memory error, because you have used up all of your 2 Gb of available memory.

B). If the "total allocated (virtual)" value is not near the possible maximum limit for your application (outlined in the section A above), but the "total available (page file)" value is very low: you have a real out of memory error. It is not because you have exhausted all free memory in your application, but it is because your system does not have enough memory available.

For example:

As you can see, while the "total allocated (virtual)" value (833.80 Mb) is far less than 2 Gb for a 32-bit app, but the paging memory has run out ("total available (page file)" is very low).

• For example, if your system has 8 Gb of RAM and has swap/paging file disabled, then system will have just 8 Gb of memory for all applications. It means no application on such system can allocate more than 8 Gb - which is far less of 8/128 Tb of possible max for 64-bit app.
• Additionally, if other applications would occupy 7 Gb of memory already, your app would have just 1 Gb to run - which is less than even 2 Gb of possible max for 32-bit app.
• Finally, a system may have swap/paging file, but its size may be limited by some value. Say, a system can have 8 Gb of RAM with 16 Gb of swap/paging file. The size of the swap/page file could be limited by system's settings or by simply running out of a free HDD space. Again, maximum memory size for all apps will be limited to 8 + 16 Gb, and will be less for your own app - depending on how many memory other apps have allocated.

Specifically, the total amount of available memory for the system from the sample error message above is 1 Gb of RAM with 512 Mb of swap/page file, resulting in 1.5 Gb of total available memory.

So you may hit this limit, and while the "largest available" value would still show the largest continuous free block in your address space to be sufficiently large (or even huge for 64-bit apps), but the system doesn't have free memory to allow your app to actually allocate more memory.

C). If the "total allocated (RTL)" value has two numbers and first number is significantly smaller than the second number: you have out of memory error because of the overhead from EurekaLog.

For example: 

Allocating: 1112 (1.09 Kb)
Largest available: 1183744 (1.13 Mb)
Total allocated (RTL): 373146095 (355.86 Mb) (2072498176 (1.93 Gb))
Total allocated (virtual): 2072498176 (1.93 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (page file): 38592802816 (35.94 Gb)

The second number in "total allocated (RTL)" (1.93 Gb) is significantly larger than the first number (355 Mb) - which means your code allocated a lot of small memory blocks (objects), so now you have a large overhead (about x5.4 times). In other words, you have out of memory error because of additional overhead from EurekaLog memory/leaks checks.

On the other hand:

For example:

Allocating: 1183744 (1.13 Mb)
Largest available: 867882 (847.54 Kb)
Total allocated (RTL): 686788239 (654.97 Mb)
Total allocated (virtual): 1086788239 (1 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (page file): 38592802816 (35.94 Gb)

As you can see, there is plenty of free memory available in your application, because the "total allocated (virtual)" value (1 Gb) is not reaching the limit (2 Gb for a normal 32-bit app, see section A above) and the "total available (page file)" value is not hitting the system's limit (see section B above). However, the "largest available" value is very small - smaller than the requested memory (1.13 Mb). It is the classic symptom of a memory fragmentation: you try to allocate a (large) block and you can't, even though you appear to have enough free memory.

F). If the "largest available" value is more than few Mb in size and is significantly larger than the "Allocating" value, while the "total allocated (virtual)" value is not close to the maximum possible limit (see section A above) and the "total available (page file)" value is large enough (see section B above): you probably do not have an out of memory error at all. 

You see, memory allocation in Delphi (Builder) is implemented like so:

function GetMem(Size: NativeInt): Pointer;
begin
  if Size = 0 then
    Exit(nil);
  Result := MemoryManager.GetMem(Size);
  if Result = nil then
    Error(reOutOfMemory);
end;

As you can see, any memory allocation error is treated like "out of memory" error. So, what this error is saying is "unable to allocate memory" for whatever reason. It could be due to memory corruption bug somewhere. For example, a memory bug could damage control structures of the memory manager.

For example:

Allocating: 1112 (1.09 Kb)
Largest available: 1183744 (1.13 Mb)
Total allocated (RTL): 1193750159 (1.11 Gb)
Total allocated (virtual): 2076430336 (1.93 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (page file): 38592802816 (35.94 Gb)

Notice while the "largest available" value (1.13 Mb) is larger than the "allocating" value (1.09 Kb), but it is not significantly larger.

The reason the "largest available" value is larger than the "allocating" value is because your application does not allocate memory directly from the system. Instead, your allocation requests go to your app's memory manager, which is the FastMM by default (on modern IDEs). Memory manager allocate larger chunks from the system and then uses these chunks to serve memory allocation requests from your code. Specifically, FastMM uses chunks of about 1.2 Mb in size. As you can see the available free memory (1.13 Mb) is less than 1.2 Mb required for FastMM to allocate a new chunk. Therefore, this case is a real out of memory error.

Another example:

Allocating: 2097152 (2 Mb)
Largest available: 1183744 (1.13 Mb)
Total allocated (RTL): 1193750159 (1.11 Gb)
Total allocated (virtual): 2076430336 (1.93 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (page file): 38592802816 (35.94 Gb)

Notice the "largest available" value (1.13 Mb) is smaller than the "allocating" value (2 Mb). Therefore, this case is a real out of memory error.

On the other hand:

Allocating: 1112 (1.09 Kb)
Largest available: 138109829251072 (125.6 Tb)
Total allocated (RTL): 3522839902 (3.28 Gb)
Total allocated (virtual): 4914884608 (4.58 Gb)
Total available (page file): 20763758592 (19.34 Gb)
Total (page file): 38592802816 (35.94 Gb)

Notice the "largest available" value (125.6 Tb) is larger than few Mb and it is significantly larger than the "allocating" value (1.09 Kb), while the "total allocated (virtual)" is nowhere near close to the possible max and the "total available (page file)" indicate plently of free swap/page file space. All of this means: there is plenty of free memory in application, but request to allocate memory fails. Therefore, it is not a real out of memory error. It is a memory corruption error.

Solution:

A). Insufficient system's memory.

If you have insufficient system's memory (not enough RAM or/and swap/page file) - fix it. Enable swap/page file (if disabled), increase size of the swap/page file, and/or install more RAM.

You may follow other advices below (especially if you have a large overhead from EurekaLog), but increasing your system's memory is a faster and easier way.

B). Memory corruption.

If you have a memory corruption error - start searching for it.

We would recommend to enable memory and leaks checks in EurekaLog. Just be sure leaks checks are enabled (for example, you don't run your app outside debugger while having the "Active only when running under debugger" option enabled).

If it did not help - try to use full-featured debugging memory manager. Such as FastMM in full debug mode, SafeMM, etc.

If it did not help - try to review your code.

C). Memory fragmentation.

Memory fragmentation is when most of your memory is allocated in a large number of non-contiguous blocks, or chunks - leaving a good percentage of your total memory unallocated, but unusable for most typical scenarios. If you have a memory fragmentation - start by reviewing your code. 

First, ensure your code does not leak. Even a smallest leak of just 1 byte being repeated enough times can occupy your free memory dividing it into smaller blocks, thus causing fragmentation. Therefore, check your application for memory and resource leaks. Fix all leaks that you can find.

Second, once you are sure there are no leaks, and you left with pure memory fragmenation: review your code. Memory fragmentation is caused by the behavior of an application, it is about greatly different allocation lifetimes. In other words, some blocks/objects are allocated and freed regularly while other persist for long periods of time. The best thing you can do to address this type of a problem is logically divide objects by lifetimes. Basically, you want to have two (or more) heaps (pools) of temporal and permanent blocks/objects, or blocks/objects of a similar life-time. For example, if your application opens a file/document, you may want to allocate everything related to that file/document in a standalone heap/pool.

Additionally, you can reduce memory fragmentation by reducing the amount you allocate / deallocate. For example, you can use dynamic arrays as pool of various things, so instead of allocating a million of small objects - it is better to allocate a single array of small records. See also the "C). Out of memory" section below.

Notice that you can override the .NewInstance method for some classes to allocate objects differently: say, on an array or via specific heap.

Finally, you may try to replace your memory manager with another one that offer low fragmentation allocations.

Generally you don't need to worry about memory fragmentation, unless your application is long-running and does a lot of allocation (and freeing). It starts to be a problem when you have a mix of short-lived and long-lived blocks/objects on a long distance. Therefore, if you have such application (such as service) - you may be able to restart it from time to time, if it is possible - thus avoiding memory fragmentation error in the first place.

D). Out of memory due to overhead.

Running out of memory because of EurekaLog's overhead means your code allocates a lot of small memory blocks (objects).

If you have out of memory error because of added EurekaLog's overhead:

1. If you need/want EurekaLog's features of memory and/or leaks checks: you can follow advices from the "E). Out of memory" section below to reduce memory footprint of your application and/or you can follow the "C). Memory fragmentation" section above to refactor your code to allocate memory more efficiently (less allocations, larger blocks);
2. If you don't need/want EurekaLog's features of memory and/or leaks checks: you can disable leaks checks in EurekaLog by unchecking the "Catch memory leaks" option. If that did not help - you can disable memory checks in EurekaLog by unchecking the "Enable extended memory manager" option.

E). Out of memory.

You have out of memory error, but it is not because of EurekaLog's overhead. For example, your application allocates 1.8 Gb of memory and runs just fine, but if you add large enough code to your application (not just EurekaLog, any code of a sufficient size) - your application may not fit inside 2.0 Gb. If this is the case, you have no other choice but to decrease memory footprint of your application.

1. Enable the "large address aware" PE flag, which will give your 32-bit app an extra 1 Gb of memory on 32-bit systems and extra 2 Gb of memory on 64-bit systems:
   1. You can use "Project" / "Options" menu to open project's options, then enable the "Building" / "Delphi Compiler" / "Linking" / "Enable large addresses" option on modern IDEs;
   2. You can add {$LARGEADDRESSAWARE ON} directive on modern IDEs;
   3. You can add the {$SetPEFlags IMAGE_FILE_LARGE_ADDRESS_AWARE} directive if your IDE does not support {$LARGEADDRESSAWARE ON} (don't forget to include the Windows unit for the declaration of the IMAGE_FILE_LARGE_ADDRESS_AWARE).
2. Ensure the "Reserve lower memory" option is turned off.
3. Use a memory profiler to find what is occupying most of your memory. Start with largest ones.
4. Do not load all information into memory at the same time. Load necessary information only when required and unload immediately after use. For example, do not use auto-create forms, create forms on demand. Pay attention to scopes of your objects and things with automatic reference counting (strings, arrays, interfaces, etc.).
5. Try to offload large information blocks into files or database. For example, giant arrays should be converted to database.
6. You may try to execute memory hungry tasks in stanalone processes, designed only to run this specific task and nothing else.
7. Offload large resources from executable into standalone files or a database. Smaller executables will occupy less address space. 
8. If you are using background threads: consider doing things in sequence rather than at the same time.
9. Review your code for memory optimization. For example:
   1. Check compiler's optimizations; 
   2. Review your code for duplicates. Pay attention to generics to avoid blowing up your code. Smaller executables will occupy less address space;
   3. Check usage of virtual functions. Ensure your application will not be dragging unnecessary code that is never called;
   4. Use algorithms with lower memory usage;
   5. A missing const in arguments could cause unneccessary copying;
   6. Use smallest possible data types and pack everything in records (however, this will probably make your application slower);
   7. Check if you have too much function inlining in your code.
10. If you have out of memory error because of a 3rd party code - try to find what specific code actually allocates that much memory. 

Finally, you can switch to the 64-bit.

See also:

• My application runs (very) slowly with EurekaLog
• How to enable/disable EurekaLog at run-time?
• How to enable EurekaLog for production, disable for development?
• How to use different settings for development/production?

I get error report about multiple leaks after using sending with Mantis send method. There is no leak report if I don't click on "Send report".

Reason:

It's a bug in Delphi's SOAP implementation.

Solution:

1. Use the latest RAD Studio version which fixes this bug.
2. Manually edit WSDLLookup.pas file and recompile it. Recompile EurekaLog to use new file (this will require EurekaLog Enterprise edition with full source code).

Update: 

Use latest EurekaLog version, we are not using Delphi's SOAP implementation anymore.

Even empty C++ Builder application crashes with "Abnormal program termination" or system crash dialog at exit.

Explanation:

Your project has memory or resource checks enabled. However, you did not configured your project correctly. Memory/resource finalization calls are in the wrong order.

Solution:

You can either:

a). Disable memory and resource checks;

or

b). Configure initialization order of your project.

See also:

• Leaks in empty C++ Builder application

I receive leak reports from EurekaLog in a blank empty VCL application.

Reason:

1. You did not set up your project properly, so EurekaLog's leak checking code is running too early.
2. You may see technical leaks. See below for an example.

Example (technical leak):

1. When application exits: RTL enters shutdown cycle inside _cleanup() routine from initexit.c file. This routine will call any registered finalization subroutine (e.g. exit procedure), such as EurekaLog and I/O streams finalization.
2. C++ RTL will flush opened streams during shutdown (fflush() routine from fflush.c file is called from _exit_streams() finalization routine from streams.c file).
3. Unfortunately, flushing stream will keep an allocated memory block used for stream locking (lock is allocated from _lock_stream() from streams.c).
4. This memory block will remain alive, even when other code (including EurekaLog) cleanup (inside cleanup cycle in the _cleanup() routine).
5. EurekaLog will detect memory block that was allocated, but not freed. EurekaLog will report this block as leak.
6. Once EurekaLog (and other code) finishes shutdown - RTL will continue to cleanup.
7. RTL will delete allocated locks by calling _cleanup_handle_locks() and _cleanup_stream_locks() (after _cleanup() routine completes) and terminate the process.

Solution: 

1. Follow our guidelines for setting up your project for leak reporting. Basically, you need to ensure that EurekaLog's code is initialized first and finalized last. Otherwise, EurekaLog's leak checking code will run when there are unallocated resources. Such resources will be reported as leaks, even though it is finalized later.
2. Ignore technical leaks. You may try to add such leaks as expected leaks. Or you may try to cleanup such leaks manually before execution of EurekaLog's leaks checking. E.g. calling routines such as _cleanup_stream_locks() at shutdown. But this may not always be possible.

See also:

• I get "Abnormal program termination" or other type of crash at shutdown