Memory Violation Part I

Purpose of Document

Note: If you encounter business function error while using standard application or UBE, it maybe already reported, install ESU for fix. If issue has not been reported, open service request with team that support the application or UBE, example, select product JD Edwards EnterpriseOne Sales Order Management while opening a SR if encountering business function issue while using P4210.
If you encounter issue with standard business function used in custom object (application or UBE), check if issue also appear with standard application or UBE. If issue does not appear, most likely there is no bug with the standard business function.

Memory Error
Memory Leak
Memory Corruption
Components for Source File (.C File)
Memory Diagnostic Tools in EnterpriseOne:
Case Study in Interactive Application:
Memory violation in Batch Application:
Open Table Not Closed:
Function Cleanup:
JDB_FreeEnv()
JDE INI Parameters which ease you to debug:
Other issues - Max Cursor and Max Data Pointer:
Windows Heap Manager
Other Resources for Troubleshooting memory issue
Troubleshooting additional
Memory Corruption - Common Mistakes
Summary:
Known Issues and Bugs:

The purpose of this document is to explain in high level, some memory issues encountered in running standard business functions through the CallObject Kernel in EnterpriseOne. This document includes case studies and resolutions on certain memory issues. This document is not intended to cover specific definition on memory errors or how to detect memory problems.

This document is intended to help internal JDE developer on what to check when hitting memory errors in EnterpriseOne and how to apply possible fix for these errors while working on reported bug. But information in this document maybe helpful for developing custom business function.

The audience for this document is someone with expert 'C' Programming Language development knowledge. This document only contains examples. It is not meant to be all encompassing. Also see the second document in this series document 1554611 Case Study on Memory Violation, Memory Corruption, Memory Overwrite, ACCESS VIOLATION and Zombie Kernel Caused by Standard Business Functions Part 2.

This document does not cover any memory corruption issues caused by the Tools Foundation Layer which is usually fixed through tools release upgrade. This document mainly focuses on memory errors caused by standard business functions which results in CallObject Kernels to crash or runbatch application to fail.

Caution: This document may contain information, software, products or services which are not supported by Oracle Support Services and are being provided ‘as is’ without warranty. Please refer to the following site for My Oracle Support Terms of Use: https://support.oracle.com/CSP/ui/TermsOfUse.html. For custom business function, field services may need to be engaged.

In running EnterpriseOne applications, memory error may occur in 3 different locations,

Register memory on Logic Server (where logic, or business function is running)
Virtual memory space in JAS (Java Application Server which calls logic)
Virtual memory area where image of applications is running

In this document, we will be focusing on memory errors in the register memory on the logic server. We will concentrate on C programming or Business Functions in EnterpriseOne.

In EnterpriseOne, memory can be represented as data structure, cache, handles (or pointers), cursor, data pointer and so on. Additionally, this document explains some case studies based on ill-treatment of memory in EnterpriseOne Business Functions. Since C's pointers give great possibility on memory manipulation, in return which may cause some memory problem. Memory corruption in client machine may cause service or application getting terminated.

Memory Error

For easier reading, we shall categorize memory error into Memory Leak and Memory corruption,

Memory Leak: Memory is allocated but not released/freed/terminated.
Memory corruption: Memory area or, buffer has altered unintentionally (without an explicit assignment). Usually unexpected altering of data stored in memory or the altering of a pointer to a specific place in memory

Note:

Memory Violation: Violating memory safety which in turn returns unexpected result
Memory/Buffer Overwrite: Writing more than allocated memory
Zombie Kernel/Process: Memory which is live but is not being referenced by any other user/applications

Memory Leak

Memory leak takes place when allocated memory is not released/terminated/killed/flushed/deleted. Reducing the available memory for other applications and eventually causing the system to page virtual memory to the hard drive slowing the application or crashing the application when the computer memory resource limits are reached. The system may stop working as these limits are reached. From Tools Release 8.97 onwards, Kernel Recycling will happen when it is not executing any BSFN for a certain amount of time.
This is implemented only for Call Object kernels. In EnterpriseOne (CallObject) Kernel Recycling can free leaked memory.

Un-paired API calling

For further detail, please refer to the "JD Edwards EnterpriseOne Tools 9.1 Development Tools: APIs and Business Functions Guide".

Database APIs (Understanding Database Communication Steps)
Several APIs called in succession can perform these steps for database communication:

Initialize communication with the database.
Establish a connection to the specific data to access.
Execute statements on the database.
Release the connection to the database.
Terminate communication with the database.

This table lists some of the API levels and the communication handles and API names that are associated with them:

API Name	Communication Handles	API Level
JDB_InitEnv	Environment handle	Control level (application)
JDB_InitUser	User handle (created)	Control level (application)
JDB_InitBhvr	User handle (retrieved)	Request level (business function)
JDB_OpenTable	Request handle	Request level (business function)
JDB_FetchKeyed()	Request handle	Request level (business function)
JDB_CloseTable	Request handle	Request level (business function)
JDB_FreeBhvr	User handle	Request level (business function)
JDB_FreeUser	User handle	Control level (application)
JDB_FreeEnv	Environment handle	Control level (application)

Note:

Control Level provides functions for initializing and terminating the database connection.
Request Levelprovides functions for performing database transactions. The request level functions perform these tasks:
- Connect to and disconnect from tables and business views in the database.
- Perform data manipulation operations of select, insert, update, and delete.
- Retrieve data with fetch commands.

HENV The environment handle contains information related to the current database connection and valid connection handles. Every application connecting to the database must have an environment handle. This handle is required to connect to a data source.
HUSER: The user handle contains information related to a specific connection. Each user handle has an associated environment handle with it. A connection handle is required to connect to a data source. If you are using transaction processing, initializing HUSER indicates the beginning of a transaction.
HREQUEST The request handle contains information related to a specific request to a data source. An application must have a request handle before executing SQL statements. Each request handle is associated with a user handle.

JDECACHE and JDECACHE APIs

JDECACHE is a component of JDEKRNL that can hold any type of indexed data that the application needs to store in memory, regardless of the platform on which the application is running; therefore, an entire table can be read from a database and stored in memory. No limitations exist regarding the type of data, size of data, or number of data caches that an application can have, other than the limitations of the computer on which it is running. Both fixed-length and variable-length records are supported. To use JDECACHE on any supported platform, you need to know only a simple set of API calls.
Data handled by JDECACHE is in RAM. Therefore, ensure that you really need to use JDECACHE. If you use JDECACHE, design the records and indices carefully. Minimize the number of records that you store in JDECACHE because JD Edwards EnterpriseOne software and various other applications need this memory as well.
JDECACHE supports multiple cursors, multiple indexes, and partial keys processing. JDECACHE is flexible in terms of positioning within the cache for data manipulation, which improves performance by reducing searching within the cache.

JDECACHE Manipulation APIs

You can use the JDECACHE manipulation APIs for retrieving and manipulating the data in the cache. Each API implements a cursor that acts as pointer to a record (user specific) that is currently being manipulated. This cursor is essential for navigation within the cache. JDECACHE manipulation APIs should be passed handles of these types HCACHE and HJDECURSOR:

Calling JDECACHE APIs

JDECHACHE APIs must be called in a certain order,
Prerequisites
Before you can use JDECACHE, you must:

Define an index: The index specifies to the cache the fields in a record that are used to uniquely identify a cache record.
Initialize a cache: Each group of data that an index references requires a separate cache.

API Calling	Explanation
1. Call JDB_InitBhvr	Each cache must be associated with a user, you must also pass the user handle obtained from the call to JDB_InitUser. This API returns an HCACHE handle to the cache that JDECACHE creates. This handle appears in every subsequent JDECACHE API to identify the cache. HCACHE: Identifies the cache that is being worked. HJDECURSOR: Identifies the position in the cache that is being worked.
2. Create index or indices
3. Call jdeCacheInit, jdeCacheInitEx, jdeCacheInitMultipleIndex, or jdeCacheInitMultipleIndexEx.	The jdeCacheInit and jdeCacheInitMultipleIndex APIs initialize the cache uniquely per user. Therefore, if a user logs in to the software and then runs two sessions of the same application simultaneously, the two application sessions will share the same cache. Consequently, if the first application deletes a record from the cache, the second application cannot access the record. Conversely, if two users log in to the software and then run the same application simultaneously, the two application sessions have different caches. Consequently, if the first application deletes a record from its cache, the second application will still be able to access the record in its own cache. The jdeCacheInitEx and jdeCacheInitMultipleIndexEx APIs function exactly the same, but they additionally enable you to define the maximum number of cursors that can be opened by the cache. The jdeCacheInitUser and jdeCacheInitMultipleIndexUser APIs initialize the cache uniquely per application. Therefore, if a user logs in to the software and then runs two sessions of the same application simultaneously, the two application sessions will have different caches. Consequently, if the first application deletes a record from its cache, the second application can still access the record in its own cache.
4. Call jdeCacheAdd
5. Call jdeCacheOpenCursor	a JDECACHE cursor is a pointer to a record in a users cache
6. Call JDECACHE Operations	JDECACHE operation APIs can be called any order
	Mainpulation of Cache: jdeCacheOpenCursor, jdeCacheResetCursor, jdeCacheAdd, jdeCacheFetch, jdeCacheFetchPosition, jdeCacheUpdate, jdeCacheDelete, jdeCacheDeleteAll, jdeCacheCloseCursor, jdeCacheFetchPositionByRef, jdeCacheSetIndex, jdeCacheGetIndex etc.
jdeCacheTerminate	For every jdeCacheInit, jdeCacheInitEx, jdeCacheInitMultipleIndex, or jdeCacheInitMultipleIndexEx, a corresponding jdeCacheTerminate must exist, except instances in which the same cache is used across business functions or forms. In this case, all unterminated jdeCacheInit, jdeCacheInitEx, jdeCacheInitMultipleIndex, or jdeCacheInitMultipleIndexEx calls must be terminated with a jdeCacheTerminateAll. A jdeCacheTerminate call terminates the most recent corresponding jdeCacheInit or jdeCacheInitEx. This means that the same cache can be used in nested business functions. In each function, perform a jdeCacheInit or jdeCacheInitEx or jdeCacheInitEx that passes the cache name. Before exiting that function, call jdeCacheTerminate. This does not destroy the cache. Instead, it destroys the association between the cache and the passed HCACHE handle. The cache is completely destroyed from memory only when the number of jdeCacheTerminate calls matches the number of jdeCacheInit or jdeCacheInitEx calls. In contrast, one call to jdeCacheTerminateAll destroys the cache from memory regardless of the number of jdeCacheInit, jdeCacheInitEx, jdeCacheInitMultipleIndex, or jdeCacheInitMultipleIndexEx calls or jdeCacheTerminate calls

Memory Corruption

Memory Corruption can occur when :

Buffer overflow

Overwrite beyond allocated length - overflow
Index of array out of bounds (array index overflow (index too large) or array index underflow (negative index))

Using an address (a pointer or handle) before memory is allocated and set
Using a pointer which is already freed
Freeing/Deleting/Terminating memory/cache which has already been freed.
Pointer persistence: Function returning a pointer from the stack which can get overwritten by the calling function
Incorrect passing of a function argument: In general, through dependency problem
Un-paired API calling

Note:

The cause of memory corruption may vary. Therefore it is important to review logs (JDE_*.logs, JDEDEBUG_*.logs, Dump Log and Call Stack).
Check e1root_.log in JAS
Check JDE_*.log for SAW Kernel and JDENET_n kernel which contains offending business function with victim

Memory Corruption and Example:

For further detail, please refer to the "JD Edwards EnterpriseOne Tools 9.1 Development Tools: APIs and Business Functions Guide" (Link JD Edwards EnterpriseOne Documentation)

According to industry terms , memory corruption can be categorized as follows:

Cause	Explanation	Example	Resolution	Others
Memory is not initialized	Contents of uninitialized memory are treated as garbage values. Using such values can lead to unpredictable program behavior.	Refer to components for c Source file below. Initializations (of variables)	Correct code.
Memory belongs to different process	It is common to use pointers to access and modify memory. If such a pointer is a null pointer, dangling pointer (pointing to memory that has already been freed), or to a memory location outside of current stack or heap bounds, it is referring to memory that is not then possessed by the program. Using such pointers is a serious programming flaw. Accessing such memory usually causes operating system exceptions, also termed page faults, which most commonly lead to a program crash.	Since EnterpriseOne CallObject Kernel is running in multi-thread so it is rare to have this type of issue.	Move global variables to parameter of calling/called business functions	This definition is more for C/C++ language in general rather than EnterpriseOne. In EnterpriseOne most of business function holds general information as parameters which minimize the usage of Globals.
(Memory) Buffer Over flow (Using beyond allocated Memory)	If an array is used in a loop, with incorrect terminating condition, memory beyond the array bounds may be manipulated. Buffer overflow is one of the most common programming flaws exploited by computer viruses causing serious computer security issues (e.g. return-to-libc attack, stack-smashing protection) in widely used programs. One can also incorrectly access the memory before the beginning of a buffer.	This issue can come in a single string (which is character array) in stripping a certain character. Or in running R3483 the allocated memory is far smaller than cache to be created.	Allocate correct size of array.	Or buffer overrun
Incorrect Heap Memory Management	Memory leaks and freeing non-heap or un-allocated memory are the most frequent errors caused by faulty heap memory management	Refer to heap and stack section. Usually resulted in ACCESS VIOLATION	N/A	Refer to http://msdn.microsoft.com/en-us/library/ms810603.aspx

Note

This category is borrowed from outside JDE which is generally recognized in C Programming
The result of memory corruption is Zombie Process (or Zombie Kernel) because memory corruption shall failed to calling routine
In iSeries platform JDE log shall contain CallStack information
(In current tools release) windows platform dump memory corruption
JDEDEBUG.log may ends abruptly (If transaction ends successfully CallObject kernel log contains calling of business function jdeFreeEnvBSFN
It is important to capture jdedebug.log and analyze it where you can check the function then focus on the last 2 functions,
- Case (corruption is caused by calling routine) Example 1
- Case (corruption is caused by called routine) Example 2

Typical Business Function components in EnterpriseOne

For example (B4200310.c),

JDEBFRTN(ID) JDEBFWINAPI F4211FSBeginDoc(LPBHVRCOM lpBhvrCom, LPVOID lpVoid, LPDSD4200310H lpDS)
{
:
:
}

Above statement can be understood F4211FSBeginDoc is Windows API which can be called directly from application and returns (ID - pointer) with 3 parameters. All business functions share the same return type and parameter data types. Only the function name and the data structure number vary between business functions.

JDEBFRTN(ID)JDEBFWINAPI: All business functions will be declared with this return type. It ensures that they are exported and imported properly.
Parameter names (lpBhvrCom, lpVoid, and lpDS) will be the same for all business functions.
LPBHVRCOM: Pointer to a data structure used for communicating with business functions. Values include an environment handle.
LPVOID: Pointer to a void data structure. Currently used for error processing; will be used for security in the future.
LPDS4200310H: Pointer to a data structure containing information that is passed between the business function and the application or report that invoked it. This number is generated through Object Librarian.

Component	Where created (repository)	Detail
Business Function Specification	OMW - Business Function Design (Central Object in Database Server)	F98762 - JDEBLC - Behavior Information JDEBLC
Data Structure Specifications	OMW - Data Structure Design (Central Object in Database Server)	F98743 - Data Structure Templates DSTMPL
.H File	Generated in Business Function Design Modified with the IDE (Deployment Server)	Header File Comment Table Header Inclusions External Business Function Header Inclusions Global Definitions Structure Type Definition DS Template Type Definition Source Preprocessor Business Function Prototype Internal Function Prototype
.C File	Generated in Business Function Design Modified with the IDE (Deployment Server)	Refer below "Components for Source File"

Components for Source File (.C File)

components	Usage	Example
Variable declarations	(via IDE) Declares variables that are local to the business function	ID idReturnValue = ER_SUCCESS;
Declare structures (Optional)	(IDE) Declares local data structures to communicate between business functions, internal functions, and the database.
Declare Pointers	(IDE) declare pointers
Check for NULL pointers	(Business Function Standard) Verifies that all communication structures between an application and the business function are valid	(lpBhvrCom == (LPBHVRCOM) NULL) \|\| (lpDS == (LPDSD4200310H) NULL))
Set Pointers	(IDE) Declares and assigns appropriate values to pointers
Initialize Behavior Routine
Initializations (of variables)
Initialize Data Structures	Allocate memory for declared structure above.
Initialize Behavior Routine		JDB_InitBhvr(lpBhvrCom, &hUser,(JCHAR *) NULL, JDEDB_COMMIT_AUTO);
Set Up Cache Index Keys	This routine can come after or before main
Main Processing	jdeCallObject(), CALLIBFRET(), CALLBF() (IDE) Provides main functionality for a business function In C standard main() is to be handled here
Function Clean Up	(IDE) Frees any dynamically allocated memory

Note:

An integrated development environment (IDE) is a software application that provides comprehensive facilities to computer programmers for software development. An IDE normally consists of:
- a source code editor
- build automation tools
- a debugger
E1: ENV: Overview of Specification (SPEC) in EnterpriseOne (Doc ID 1142313.1)
Same information can be found at C Business Functions from Oracle Documentation Library in "API and Business Function Design" for EnterpriseOne tools release

Memory Initialization

In a certain compiler it may write Warning Message C6001: Using uninitialized memory <variable>

Example:

Using uninitialized memory dsD4200370B
*** Break In Code B4200370 ***
/**********************************************************************
* Business Function structures
**********************************************************************/
DSD4200370B dsD4200370B; /* F41021 Delete Cache */
:
:
:
if (I4200310_DoDeferredCommitments(lpds4200310D, lpdsF42UI11) == TRUE)
{
I4200310_ProcessDeferredCommitments(lpBhvrCom, lpVoid, lpDS);
dsD4200370B.cMode =_J('2');
}

Solution:

The solution is to initialize the memory either at declaration time or with a memset.
DSD4200370B dsD4200370B = {0};
--- or ---
memset((void *)&dsD4200370B, 0x00, sizeof(dsD4200370B));

Global Variables (Memory belongs to other process)

Since EnterpriseOne CallObject Kernel can be running in multi-threaded CallObject Kernel Process. So it is very unlikely to have this type of issue in the EnterpriseOne release you are working on. For detail refer to <Document 748330.1> Multithreaded Kernels.

Currently business functions in EnterpriseOne are threadsafe (that is, can safely share a kernel with other business functions). In general, to make your business functions threadsafe, you must remove global and static variables, as
well as references to non-threadsafe APIs.

GLOBALS

Often, global structures are used so that each function does not need to declare a local instance. For example, they may be used as a mechanism to pass information between functions. In this case, pass in the value as a parameter instead of using a global, because information on the stack is thread-specific and, thus, threadsafe. If it is impossible to pass the value via a parameter, consider storing and retrieving the value from JDECACHE. However, keep in mind the fact that you can degrade performance if you store a very large quantity of redundant information in cache or if you access cache a very large number of times.
" The global is used as a constant. In this case, do not declare a variable; instead, use #define or other constructors.
" The global is used as an implicit parameter. In this case, change the function prototype to accept a new parameter.
" The global is extraneous or used as a local variable only. In this case, reduce the variable scope as appropriate.

STATICS

Static variables are often used to store state data, such as an ID counter or a last ID used reference. However, state information should be stored within JDECACHE. Static variables are also used to cache information created just once. Instead, use local variables for this function. If a global is used to specify a default value or constant, be sure to declare the global as 'const' to ensure that no threads can modify the value.
" The static is used as a constant. In this case, replace it with a local constant.
" The static is used as an implicit parameter. In this case, change the function prototype to accept a new parameter.
" The static is not used to store state information. In this case, remove the keyword (static) from the variable declaration.

Buffer overflow

Buffer overrun or buffer overwrite can be used to explain this terms.

C6386: Buffer overrun accessing <buffer>, the writable size is <write-size>, but <write-index> bytes may be written

Example:

Buffer overrun: accessing lpDS->szDescription the writable size is 62 bytes but 64 bytes might be written.
JCHAR szDescription[31];
ushort nMaxSize = 31;

nMaxSize = DIM (szDescription);
while (lpDS->szLevelOfIndention[nCounter] == _J('.'))
{
for (j = (nMaxSize - 1); j > 0; j--)
{
szDescription[j] = szDescription[j-1];
}
szDescription[0] = _J(' ');
++nCounter;
}/*end while*/
szDescription[nMaxSize] = _J('\0');
DIM will return the length of the szDescription variable, which is 31. Since the C Programming language is a zero origin language, the valid indexes are 0 to 30. The assignment above is using 31 which is outside the bounds of our string.

Solution:

Modify the code to stay within the boundaries of the string. In this case one can be subtracted from the nMaxSize variable because we know DIM returned the length of the variable, not the length of what is in the variable.
szDescription[nMaxSize-1] = _J('\0');

Incorrect Heap Manager

JDEdwards EnterprisneOne do not have control over Heap manager. Refer to your vendor for detail because this is very much operating system dependent. Some external reference:-

Windows Heap Memory Manager: http://msdn.microsoft.com/en-us/library/ms810603.aspx
iSeries Heap Memory Manager: http://publib.boulder.ibm.com/infocenter/iseries/v7r1m0/index.jsp?topic=/rtref/stacktracebacks.htm
SUN Solaris: http://developers.sun.com/solaris/articles/solaris_memory.html

Heap Memory Error and Stack Memory Error:

Cause of Heap memory errors and Stack memory errors,

Memory errors:

Heap memory errors:

Attempting to free memory already freed.
Freeing memory that was not allocated.
Attempting to write to memory already freed.
Attempting to write to memory which was never allocated.
Memory allocation error.
Reading/writing to memory out of the bounds of a dynamically allocated array

stack (local variables) memory errors:

Reading/writing to memory out of the bounds of a static array. (array index overflow - index too large/underflow - negative index)

Function pointer corruption: Invalid passing of function pointer and thus a bad call to a function.

Memory Segments

Memory Diagnostic Tools in EnterpriseOne:

Refer to the following sections in the JD Edwards EnterpriseOne Tools Server and Workstation Administration Guide Release 9.1 (HTML Link: http://docs.oracle.com/cd/E24705_01/doc.91/e24259/toc.htm , PDF Link: http://docs.oracle.com/cd/E24705_01/doc.91/e24259.pdf)

13.7 Troubleshooting the Enterprise Server Processes
13.7.2 Evaluating EnterpriseOne Server Performance
13.7.1 Understanding Resource Utilization and Performance

The JD Edwards EnterpriseOne kernel architecture is divided into the C business function layer and the tools foundation layer. This architecture is susceptible to the intrinsic memory manipulation weaknesses in the C programming paradigm of memory leaks and corruption. This can lead to crashed CallObject kernels and runbatch processes which directly result in failed business transactions, loss of productivity and downtime usually requiring an excessive amount of time to troubleshoot and correct.

In addition, the kernel and batch processes utilize resources and place demands on memory and the CPU and execute in the context of other processes that also demand memory and CPU resources. Each process has its own level of impact on the system, and the cumulative effect of all processes currently running will impact performance. Appropriate data can be captured and used with measurement tools to evaluate overall performance as well as the impact of individual processes. When utilization or performance exceeds certain thresholds, it is critical to have tools that can diagnose which process is creating the resource overload in order to correct the problem and restore the performance to normal levels before the system crashes.

A new administration tool called Kernel Resource Management (KRM) has been added to Server Manager to enable you to increase system stability and simplify the troubleshooting process. Graphs allow you to quickly identify processes with high resource consumption and recycling allows the ability to reclaim system resources.

There are graphs for the following:
Summary Graphs for entire Enterprise Server that displays memory and CPU usage.
At Enterprise Server level, there are graphs that display the Top 10 Processes by Memory and CPU.
At the Individual Process Level there are graphs for CPU, Memory, caches and DB connections.

New diagnostics allows you to quickly identify root cause of resource consumption issues. For Diagnostics, information about the current resource usage is written to the jdedebug.log:
Cache information:
Cache name
Number of records in the cache
Indices

Database Transactions:
Commitment type
User
Application

New tools allow the JD Edwards EnterpriseOne development and QE processes to identify and resolve resource consumption defects prior to RTM. KRM enables you to monitor kernel and batch processes and to diagnose CPU and memory usage issues.

Memory Diagnostics

Case Study in Interactive Application:

Refer to part 2 of this document: <Document 1554611.1> ^{E1: BSFN: Case Study on Memory Violation, Memory Corruption, Memory Overwrite, ACCESS VIOLATION and Zombie Kernel Troubleshooting.}

Memory overwrite: For this example, two business functions are not able to communicate properly. The size of string (Character Array) of calling routine is smaller than the size assigned by called routine.
Buffer Overflow (called Business Function fails into zombie): Called business function fails to return to calling routine. This type of memory error can occur when a certain routine is checking individual characters in a character array (namely, string)
Integer value is too long: This case is to show possible memory corruption which is caused by data. Similar issue can occur when a certain string value contains Special Character in it. For special characters in C language, refer to <Document 1271740.1> E1: BSFN: How to Write a C Business Function to Return a Special Character.
ACCESS VIOLATION: This error is coming usually in function cleanup where free memory, close handles and so on is used on 64-bit Windows Servers.
Memory Overwrite II: This type of error can come in API jdeStrcpy() and jdeStrncpy() when the size between left operand is different from right operand.
Example 1: In AS400, all the inventory transaction fails when transaction UOM is different from Primary UOM
Column Array Mismatch: The Columns array used to open the table did not match the data structure used to fetch the data from the table; as a result the fetched record from table got incorrectly mapped to the cache Data Structure in return this resulted in memory overwrite. Most of standard business function stores data pointer rather than returning actual column value but when there is any business function which has to return column value to data structure defined, the sequence of columns for a specific table (through Array) has to be in synch with data structure defined.

Memory violation in Batch Application:

Memory Violation message in PDF or JDEDEBUG.log shall not reflect actual offending business function. Because Memory Violation message in PDF is to be written by UBE Kernel, which reads and process line by line based on event rule as a result it only indicates the first level business function (which is called by report). So if a certain UBE is calling any business function through UBE Kernel, which is waiting for called business function to return. If it fails, then Memory Violation Message is to be written in PDF. For this reason if you turn on UBE log, same message can be retrieved. For this case, it is very important for you to capture both UBE kernel log and CallObject Kernel log by setting tracing (which records callobject process) and UBE Logging Level 6.

As we have describe above, Memory Violation can be understood as violating memory safety which in turn returns unexpected result. So you need to resolve this message to guarantee the result you are getting is correct and valid.

Unlike interactive application, transaction volume in a certain batch may affect result so verify that whether there is any memory leak. If so, resolve memory leak first to move on to memory corruption. Most of time, memory leak and memory corruption can be related in batch.

For detail, Refer to part 2 of this document: <Document 1554611.1> E1: BSFN: Case Study on Memory Violation, Memory Corruption, Memory Overwrite, ACCESS VIOLATION and Zombie Kernel Troubleshooting.

Open Table Not Closed:

Most commonly un-faired JDB API can cause this issue which failed to disconnect from database. But if a certain application is closed properly (either through UBE Kernel for batch job or JAS for interactive application) jdeFreeEnvBSFN will detect and determine all the memory leak and open connection. If any, this same routine will handle this type of memory leak.

For further detail, refer to JDB_FreeEnv() section

Function Cleanup:

This section is to show correct coding in "Function Clean Up" section of C Source Codes. Our case study shows that most commonly ACCESS VIOLATION is caused by below routines. But it is not madatory to check whether handle, pointer, data pointer and so on is NULL before calling Free(). So you may need to approach it case by case basis.

Source code	Actual Implemenation	Considerations
B4200310.c	Example 1: if (hRequestF4211 != (HREQUEST)NULL) { JDB_CloseTable(hRequestF4211); }
	Example 2: if (hUser != (HUSER)NULL) { JDB_FreeBhvr(hUser); hUser = (HUSER)NULL; }
XT4314Z2	Example 1 if ((dsGetPODetail.idPointerToF4311DataStructure != (ID) 0) && (lpdsF4311Detail != (LPF4311) NULL) ) { lpdsF4311Detail = jdeRemoveDataPtr(hUser, (unsigned long) dsGetPODetail.idPointerToF4311DataStructure ); if (lpdsF4311Detail != (LPF4311) NULL) { jdeFree( (void*) lpdsF4311Detail); } lpdsF4311Detail= (LPF4311) NULL; dsGetPODetail.idPointerToF4311DataStructure = (ID) 0; }
	Example 2: Free pointer to the F43121 Data Structure if ( dsGetPOReceipt.idPointerToF43121DataStructure != (ID) 0) { if (lpdsF43121 != (LPF43121) NULL) { jdeFree( (void*) lpdsF43121); } lpdsF43121= (LPF43121) NULL; dsGetPOReceipt.idPointerToF43121DataStructure =(ID) 0; }
	Examaple 3: Free pointer to the Item Branch record - i.e. F4102 if ((dsChkItemBr.idItemBranchRecord != (ID) 0) && (lpdsF4102 != (LPF4102)NULL)) { if (lpdsF4102 != (LPF4102) NULL) { jdeFree( (void*) lpdsF4102); } lpdsF4102= (LPF4102) NULL; dsChkItemBr.idItemBranchRecord =(ID) 0; }
	Example 4 if(hUser) /* or this can be written if (hUser != (HUSER) NULL) */ { JDB_FreeBhvr(hUser); }

JDB_FreeEnv()

The API JDB_FreeEnv() is to release all memories, connections to DB, Library caches and so on. Through JDB_FreeEnv(), memory leak or open table are to be handled for specific user and environment.

Below APIs can be called in JDB_FreeEnv();

In JDEDEBUG message ""Entering JDB_FreeEnv (EnvHandle 999999)" (for this example, 999999 is handle for hExe),

API	For	Example in Log
jdeSpecTerminate();
JDB_FreeCriticalSections();
psthread_term_library();
JDERT_CleanRealTimeEvents((HENV)lpExe);
JDB_DBWritePricingDetailData();	Write transaction based pricing information from cache to table.
jdeSecFreeSecList((HENV)lpExe);
CloseHhandlesForEXEINFO(lpExe);	Close all hRequests under lpExe
DLIST_Destroy(lpExe->lpF98BlobHandleList);
JDB_FreeUser((HUSER) lpUserInfo);	Free ghUser if it has the same environment as the current one.
KillDBRequest(lpRequestInfo);	Remove all requests.
DLIST_Destroy(lpExe->lpRequestList);	Destroy request linked list
jdeCacheDestroyAllUserCaches((HENV) lpExe);	Destroy the user's cache list
jdeFree(lpExe->lpBhvrCom);	Free Memory	jdeFree( lpExe->lpBhvrCom->lpObj ); jdeListDestroy( lpExe->lpBhvrCom->lpObj->OpenTableList ); jdeFree( lpExe->lpBhvrCom->lpObj->lpGlobalApp ); jdeFree( lpExe->lpBhvrCom->lpObj->lpzAppVars ); jdeFree( lpExe->lpVoid );
FreeLibrary(hEMailLibrary);	This will logoff from MAPI and close the Mail Handle
FreeLibraryCache(lpLib);
DLIST_Destroy(lpExe->hLib);	Destroy user linked list
jdeFreeRemoteEnvs(lpExe);	Free remote environments
DLIST_Delete(ghEnvList, NULL);	Remove from Env linked list and Free handle.
JDECM_CacheDestroy(lpExe->hWorkObjCache);
psthread_mutex_destroy(lpExe->hWorkObjCacheMutex);
set_ghEnv( (HENV) NULL);
PRT_TerminateLib();
jdeSecFreeUserList();		for KERNEL_NET Free user list maintained by the security server. This needs to be called before the freeing lpExe, FreeUserList needs lpExe parms for User, IPAddress, PID and it needs to write to F9312 History table
jdeFree(lpVoid);		Remove all security info.
DLIST_Delete(lpExe->lpSecurityList, (void * *) &lpVoid);
DLIST_Destroy(lpExe->lpSecurityList);	Destroy security info linked list
KillPOCache(lpExe);
KillOMAPCache(lpExe);
DLIST_Delete(lpExe->lpAuditInfoList, (void * *) &lpVoid);	Remove all Audit info.
DLIST_Destroy(lpExe->lpAuditInfoList);	Destroy audit info linked list
jdeListDestroyEmpty (&(lpExe->pEnv->pDBDSTransientCache));	Destroy the Metadata database datasource cache.
jdeFree(lpExe->pDataPtr[nLoopCount]);	Refer below JDB4100006	Loop through the data pointer array in the lpExe and log a debug message for each slot that has a non-null data pointer. This isn't necessarily a memory leak of some kind (because the pointer could have been freed without being removed from the array), but it's one of the slots used up which, over time, could cause the array to fill up. This code is just the early warning system. The information logged to the debug log should be enough to track back to see where the pointer was stored. If not, the user can turn on Data Pointer Tracing in the JDE.INI and watch for the problem again.
jdeFree(lpExe);
JDERT_CleanRealTimeLegacyGlobals();	Clean the legacy real time event globals
FreeLibrary(ghLib);
DestroyGlobalPricingData();
JDB_UnloadTriggerLibrariesPrv(lpEData);	Unload Table Trigger libraries
KillDBCache(lpEData, TRUE);
DLIST_Delete(ghEnvData, (void * *) &lpEData);
DLIST_Destroy(ghEnvList);
KillDriverEnv(lpDriverEnv);	Remove all DriverEnv
DLIST_Delete(ghDriverEnvList, (void * *) &lpDriverEnv);
DLIST_Destroy(ghDriverEnvList);
jdeFree(lpConnectInfo);	Remove all connectinfo.
DLIST_Delete(ghConnectInfoList, (void * *) &lpConnectInfo);
DLIST_Destroy(ghConnectInfoList);	Destroy connectinfo linked list
DLIST_Destroy(ghCacheList);
DLIST_Destroy(ghUserList);
DLIST_Destroy(ghRequestList);
FreeBSFNLibraryList();	Free all the BSFN libraries in BSFN library list and then delete the BSFN library list
JDENET_FreeKernel();
FreeOCMSharedMem();
RemoveKernelData ();
jdelib_Terminate();
KillDRCache();
KillEnvironmentMapCache();
FreeLibrary((HINSTANCE)hJDEUSERLib);
jdeGetOrSetSecondaryServer(NULL, NULL);	Free list of failed over servers
FreeKernelIPCs();
jdeSpecTerminate();	destroy mutex
JDB_FreeCriticalSections();
psthread_term_library();

Note:

"JDB4100006 - DP - While freeing the JDB environment, leaked data pointer (%p), pointer handle (%d), stored from File=%ls, Function=%ls, Line=%u." lpExe->pDataPtr[nLoopCount]->pData, nLoopCount + JDB_MAX_DATAPTR, szFileName, szFunctionName, lpExe->pDataPtr[nLoopCount]->dataPtrContextData.nLine);
Log an error message about abandoned data pointers if there are some AND:

- The code was compiled in debug mode OR
- Debug logging is turned on

The point is that there may be cases where it's not a big deal to leave a couple of these data pointers around (i.e. not a growing leak, just some stored data used while the environment exists), so we don't want to log these as errors in a production environment. We only want to log them in a "debugging" type of environment where the issue can be diagnosed (debug- built code OR debug logging on).
if (nDataPointersLeaked) "JDB4100005 - Application code leaked %d data pointers which were detected when freeing environment %ls (EnvHandle %p) for user %ls. Please refer to the debug log for details. If there are not enough details in the debug log to determine the cause, set DataPointerTracing=1 and Output=FILE in the [DEBUG] section of the JDE.INI and watch for this error message again. Or, set logDPLeaksAtSignoff=TRUE in the [DEBUG] section of the JDE.INI and look for JDB4100006 error messages. This message (JDB4100005) will only appear with debug tools code or when debug logging is turned on." (nDataPointersLeaked, lpExe->szEnv, lpExe, lpExe->szUser)

Example:

Entering jdeCacheDestroyAllUserCaches
hCache:[069E6A00] Nm:[0B4300420] Recs:0 Curs:0 Hnds:1 Cchs:0 DESTROY
CAC0001036 - Unterminated cache 0B4300420 with 0 record(s), created from File=b4300420.c, Function=IB4300420_CheckCache, Line=611.
Decremented cache count to 0
CAC0001025 - Application code leaked 1 caches which were detected when freeing environment JPD910 (EnvHandle 01D99658) for user WILLIAM. Please refer to the debug log for details. If there are not enough details in the debug log to determine the cause, set Output=FILE in the [DEBUG] section of the JDE.INI and watch for this error message again. Or, set logCacheLeaksAtSignoff=TRUE in the [DEBUG] section of the JDE.INI and look for CAC0001036 error messages. This message (CAC0001025) will only appear with debug tools code or when debug logging is turned on.

JDE INI Parameters which ease you to debug:

Below information can be modified directly in ...JDEdwards\E910\system\bin32\JDE.INI or can be modified through server manager by accessing Configuration>Logging and Diagnostics. This step is important because this can be means to track down the root cause of memory issue (both Memory Corruption and Memory Leak).

Section Name	Entry /Parameter	Default Value	Available Values	Description	Others
BSFN MEMORY DIAGNOSTICS	bmdLevel	0	0 - Disabled - No Memory Logging 1 - Level 1 - Sampled Memory Logging For All Processes 2 - Level 2 - BSFN Memory Logging (After MB Threshold) 3 - Level 3 - BSFN Memory Logging (After Specified BSFN)	BMD Level: Level of diagnostics. Level 1 logs memory and CPU usage, at a samling frequency of allocations. Level 2 logs BSFN emory and CPU usage, for all BSFNs running at BSFN levels 1 and 2, after a trigger based on a memory usage. Level 3 logs BSFN memory and CPU usage, for all BSFNs and BSFN levels, after a trigger based on a memory usage combined with a specified BSFN name and BSFN level.
	allocFrequency	15000		Allocation Frequency: Number of memory allocations for memory sampling. Each samle logs memory and CPU usage. This value is only used for BMD level 1.
	memThresholdMB	100		Memory Threshold (MB): Memory threshold (in megabytes) to begin logging detailed BSFN memory usage. For BMD level 2, this is the trigger when logging starts. For BMD level 3, this is combined with a specified BSFN name and level before logging starts.
	bsfnName			BSFN Name: Business Function name. When combined with the BSFN level and memory threshold, triggers the logging of data. Used by BMD level 3 only.
	bsfnLevel	2		BSFN Level: Business Function level. When combined with the BSFN name and memory threshold, triggers the logging of data. Used by BMD level 3 only.
	enableDebug	0	0 (False) - Do Not Change Debug Logging 1 (True) - Turn On Debug Logging With BMD Level 3	Enable Debug Logging: Option to turn on debug logging. If selected, debug logging will be dynamically turned on when trigger conditions for BMD level 3 are met. This does not change the debug logging settings of JDE.INI. Used by BMD level 3 only.
DEBUG	Output	NONE	NONE - No Debug Logging FILE - Enable Debug Logging	Enable Debug Logging: Defines whether debug (jdedebug) logging is enabled. The value configured in the JDE.INI will affect every new process that starts up. Changing this value will not affect already running process	Do not set this value to FILE unless it is request by Oracle because there are many different ways to turn on logging for specific process dynamically
	DebugFile	jdedebug.log		JDEDEBUG.LOG Filename: Defines the complete path to use when creating the debug level logging file (jdedebug.log). The supplied filename will be replaced with 'jdedebug_###.log', where '###' is the operating system process identifier (PID) of the kernel process.
	JobFile	jde.log		JDE.LOG Filename: Defined the complete path to use when creating the primary logging file (jde.log). The supplied filename will be replaced with 'jde_###.log', where '###' is the operating system process identifier (PID) of the kernel process.
	JDETSFile			TS Log File: The complete path to the JDETS log file. @what's TS??
	LogErrors	1	0 - Disabled 1 - Enabled	Enable JDE.LOG: If enabled the jde.log logging file will be created. The value configured in the JDE.INI file will affect every new process that starts up. Changing this value will not affect already running processes. It is highly recommended to always enable this type of logging.	It is important to check jde.log in hitting memory error. In latest tools release which contain detailed information on memory leak and memory corruption
	ClientLog	0	0 1	Enable Client Log: This setting controls whether logging messages generated by CallObject kernel shoul be set back to requesting fat client workstation for inclusion in it's tracing logging.
	KeepLogs	1	0 - Remove logs Once Printed 1 - Keep Logs After Printed	Keep UBE Logs: When enabled log files generated by a UBE process are kept after the UBE has been printed. If disabled the log files will be deleted.
	TamTraceLevel	0	0 through 9	TAM Trace Level: The settings configures the level of logging that should be written related to TAM file access.
	ThreadTraceLevel	0	0 through 3	Thread Trace Level: Configures the level of thread related debug information.Increasing the trace level may significantly negatively impact performance.
	QKLog	0	0 - No Logging 1 - Execution Logging 2 - Debug Logging	Queue Kernel Log: This entry configures the level of logging emitted by the queue kenrel processes.
	RepTrace	0	0 - Disabled 1 - Enabled	Trace Replication: If enabled replication activity performed by the replication kernel will be traced to the debug log file.
	CloseFiles	FLUSH	0 - Flush To Disk Periodically 1 - Close After Each Write FLUSH - Flush To Disk After Each Write	Close/Flush Debug Files: There are 3 allowed values for this setting. When set to '1', the trace (debug) log file will be closed (and effectively flushed to disk) after each message is written. When set to 'FLUSH', the trace (debug) log file will be flushed to disk (but not closed) after each message is written. When set to '0', the trace (debug) log file will be flushed to disk periodically. For th ebest performrance (with trace loggign turned on), this value should be set to '0'. To ensure all trace messges are written to the debug log (with trace logging turned on) - even when a process terminates abnormally - this value should be set to 'FLUSH' or '1'. there are 2 major differences between 'FLUSH' and '1'. The first is that 'FLUSH' will have better performance than '1'. The other difference is that '1' will make it easier to view or delete the trace (debug) log file while E1 processes are still runnning (as some editors and operating systems will not allow the viewing or deletion of files in use by other processes.)
	RunBatchDelay	0		RUNBATCH Delay: Specifies the time, in seconds, that the runbatch process shall wit prior to performing any actual work. While useful for setting a delay to allow time to connect a debugger to the process, it shluld be set to zero outside of debugging activity.
	runprintDelay	0		RUNPRINT Delay: Specifies the time, in seconds, that the runprint process ahll wait prior to performing any actual work. While useful fo rsetting a delay to allow time to connect a debugger to the process, it should be set to zero outside of debugging activity.
	TransactionLog	NONE	NONE - Disabled FILE - Enabled	Transaction Log: This entry enables saving the SQL statements into the debug logs with the actual values used. If not enabled the actual values may be masked using a placeholder character such as '2'. Enabling this option may negatively impact performance.
	EVNDelay	0		EVN Kernel Delay: This entry defines a value, in seconds, that the event notification (EVN) kernel will wait after being created and prior to processing. This is typically set only wnen debugging the EVN kernel itself so should typically be left at zero.
	DumpLPDS	1	0 - Do Not Log 1 - Do Log Data Structures	Log CallObject Data Structure: This setting enables or disables the logging of the data structures passed into and returned from a business function running within a CallObjectKernel. The data will be written to the trace (debug) if enabled. Enabling this logging may reduce the performance of the CallObject process but may also provide very useful information when developing or troubleshooting business functions.
	TraceRowSecurityFetch	FALSE	FALSE - Disabled TRUE - Enabled	Trace Row Security Fetches: If enabled database fetches that violate row security will be logged to the trace (debug) log file.
	DataPointerTracing	false	FALSE - Disabled TRUE - Enabled	Log Data Pointer Activity: If enabled activity related to caching data pointers within JDB will be captured and written to the trace (debug) log file. The need to utilize this tracing is very rare and should typically not be enabled.
	CMTrace	0	0 - Disabled 1 - Enabled	Cache Manager Trace: If enabled the cache manager (JDECM) activity will be written to the trace (debug) log file.	With trace enabled, you will see similar to these in the log: JDECM_CacheFindPtrKeyed handle=087dda40 Key : ["DEPFUT4" ] JDECM_FAILED JDECM_CacheAdd handle=087dda40 Idx#1: ["DEPFUT4" ] JDECM_SUCCESS: Add rec size:436 Total mem:931556 Total recs:2088 JDECM_CacheFindPtrKeyed handle=087dda40 Key : ["DEPFUT4" ] JDECM_PASSED
	CacheInfoFrequency	0		Cache Manager Log Frequency: If non-zero this entry defines the frequency in which cache manager statistics should be written to the trace (debug) log files. If zero the information will never be written to the logs. Valid only if cache manager logging is enabled. This frequency is considered to be the number of cache manager related APIs/functions that are allowed between log statements. Enabling this tracing may negatively affect performance.
	CMTraceFilter	ALL		Cache Manager Trace Filter: This entry defines the caches that should be logged to the trace file when cache manager logging is enabled. The value may be one or more cache names separated by either a comma or semicolon. If blank or the value 'ALL' then all caches will be logged.
	maxDebugFileSize	1GB		Max size of each debug log: This setting specifies the maximum file size for debug log files. This provides debug log rollover functionality, similar to that of the JAS server. Before each debug log file reaches the specified maimum size (specified by this setting), as new log file will be created. The preceding debug log file will be renamed using the suffix "_1". If there is already a proceding debug log with the "_1" suffix, that older file will be renamed to a suffix of "2". And so on, until maximum number of preceding debug log files is reached (specified by maxDebugBackupIndex). At that point, the earliest proceding debug log file will be deleted. The minimum size for a debug log file is two lines, regardless of this settings. When this setting is zero, there will be no maximum file size for debug logs.
	maxDebugBackupIndex	1		Max number of prior debug logs: This setting specifies the maximum number of proceding debug log files to keep. This provides debug log rollover functionality, similar to that of the JAS server. Before each debug log file reaches the maximum size (specified by maxDebugFileSize), a new log file will be created. The proceding debug log file will be renamed using the suffix "_1". If ther eis already a proceding debug log with the "_1" suffix, that older file will be renamed to a suffix of "_2". And so on, until the maximum number of pecding debug log files (specified by this setting) is reached. At that point, the earliest proceding debug log file wil be deleted. When maxDebugFileSize is set to zro, this setting is ignored, since there is not maximum file size.
	logMemDiagsAtSignOff	FALSE	TRUE FALSE	Log memory diagnostics at signoff: This setting specifies whether or not memory diagnostics will be logged when a CallObject Kernel user session is ended or a UBE process ends. The memory diagnostics will be written to the jdedebug.log.
	logDPLeaksAtSignoff	FALSE	TRUE FALSE	Log Data Pointer leaks at signoff: This setting specifies whether or not Data Pointer leaks will be logged when a user session is freed in an E1 Server process. Details about the leaked Data Pointers will be written to the jde.log.
	disableContextInfo	FALSE	TRUE FALSE	Disable Contextual Information: This setting specifies whether or not Contextual Information for HUser, HRequest and DataPointer will be available in memory diagnostics or when these specific objects are reported as leaked resources in jde log or jdedebu log.
JDEHEAP ADVANCED DIAGNOSTICS ENGINE (JADE) Configuration	trackMemUsage	0	0 (False) - Do Not Begin JADE Memory Tracking 1 (True) - Begin JADE Memory Tracking At Process Start	Track JdeHeap Memory Usage: Track jdeHeap memory usage by JADE. The tracking will start with the first jdeHeap usage of the process. It will terminate when the process shuts down and when any of the JADE buttons on the Server Manager process screen are used.
	logLevel	2	1 - Summary of JdeHeap Memory Usage 2 - Summary plus BSFN-scoped Memory Detail 3 - Summary plus All JdeHeap Memory Details	Logging Level: Level of logging the JADE diagnostics. Level 1 logs a summary of BSFN-scoped pointers and of all pointers. Level 2 logs the summary, plus, detail lines for each BSFN-scoped jdeHeap usage. Level 3 logs the summary, plus, detail lines for all jdeHeap usage.
	logInterval	0		Logging Intervals (in seconds): Logging interval (in seconds) between dumping of JADE data to the debug log file. This is a minimum interval, not an exact interval. No dumping will be done if there has been no jdeHeap activity since the last dumped data. This interval-based dumping will be termiated if any of the JADE buttons on the Server Manger process screen are used.
	bsfnTriggerUseTrigger	0	0 (False) - Do Not Use Specified Conditions To Begin JADE 1 (True) - Use specified Coonditions To Begin JADE	Use BSFN Trigger to start JADE: Use BSFN Trigger to begin JADE memory tracking. The BSFN-related trigger conditions are given in this section of the INI file. When the trigger conditions are met, JADE memory tracking begins. The trigger does not cause any JADE data to be dumpted to log files.
	bsfnTriggerMemThresholdMB	100		Memory Threshold (MB): Memory threshold (in megabytes) to begin JADE tracking of jdeHeap memory usage. This can be combined with a specified BSFN name and level to trigger when tracking starts.
	bsfnTriggerBsfnName			BSFN Name: Business Function name. When combined with the BSFN level and memory threshold, triggers the JADE tracking.
	bsfnTriggerBsfnLevel	0		BSFN Level: Business Function level. When combined with the BSFN name and memory threshold, triggers the JADE tracking. Not used when set to zreo.
	bsfnTriggerEnableDebug	0	0 (False) - Do NOt Change Debug Logging 1 (True) - Turn on Debug Logging When JADE Is Triggered	Enable Debug Logging: Option to turn on debug logging. If selected, debug logging will be dynamically turned on when trigger conditions for JADE are met. This does not change the debug logging settings of JDE.INI

Note:

BSFN MEMORY DIAGNOSTICS (BMD) provides a series of methods to identify the cause of resource usage problems, such as memory leaks. Note: do not use these settings unless investigating a problem, as system performance may be affected. Also, the EnterpriseOne services must be re-started for these JDE.INI settings to take affect. For Runbatch, each new UBE will use any changed BMD values without restarting the services. All BMD logging is placed in the jdedebug.log files, even if jdedebug logging is turned off in the JDE.INI. Three levels of BMD logging are available. BMD level 1 logs CPU and memory usage for all EnterpriseOne server processes. BMD levels 2 and 3 log CPU and memory usage for those processes which run Business Functions (Call-Object kernels and UBE jobs). BMD Level 2 logs only BSFN Level 1 and BSFN Level 2. BMD Level 3 logs all BSFN levels, and also provides an option of enabling Debug logs. Lower BSFN levels include data from all higher BSFN levels that each BSFN calls.
JDEHEAP ADVANCED DIAGNOSTICS ENGINE (JADE) provides a method to identify the location of JdeHeap memory leaks. Note: do not use these settings unless investigating a "Tools-level" problem, as system performance may be affected. For memory investigations within the scope of BSFNs, use the buttons on the Server Manager process screen, instead of these INI settings. JADE tracks memory usage of all jdeHeap functions (jdeAlloc, jdeCalloc, jdeRealloc, and jdeFree). The JADE JDE.INI settings take affect whenever a process starts. They will be ignored and deactivated whenever using a JADE button on the Server Manager process screen. All JADE logging is placed in the jdedebug.log files, even if jdedebug logging is turned off in the JDE.INI. Three levels of JADE logging are available. JADE level 1 logs a summary of memory usage. JADE level 2 logs the summary, plus detail lines for each BSFN-scoped memory pointer. JADE level 3 logs the summary, plus detail lines for all jdeHeap memory pointers. There is an option to start JADE memory tracking based on specified BSFN triggers, and also an option to turn on Debug logging when the BSFN trigger is met. JADE data is dumped to the log file based on a minimum time interval. The INI-initiated JADE tracking is terminated whenever a JADE button on the Server Manager process screen is used.

Other issues - Max Cursor and Max Data Pointer:

Rarely, in running EnterpriseOne application, memory leak can be causing unexpected result as written in <Document 1325310.1> E1: BSFN: Case Study - How to Interpret Error CAC0001017 in jde.log? The cause of this type of message is it has reached the maximum value which EnterpriseOne can hold.

Max data pointer which DB can handle is 1000

#define JDB_MAX_DATAPTR 1000
Defined at system/common/jdekrnl/jdb/jdbprv.h (encapsulated code)

Max Cursor (temporary repository of data) is 100 (JDECACHE definition),

#define JDECACHE_MAX_CURSORS 100
This file can be accessed system\include\JDEKDFN.h

JDECM_CacheInit can open maximum cursors up to 100 (defined at system\include\jdekprto.h)
You MUST NOT change any default value listed here
So data pointer error will come after 1000 open pointer and open cursor error will come after 100 cursor opened

Note:

In case your JDE.log contains above message, capture jdedebug.log after settings JDE.INI parameter DataPointerTracing=TRU in [DEBUG] section and analyze it. Since this type of error is coming in long running reports or application which huge detail lines, limit the size of debug by setting maxDebugFileSize=1GB in same section.
If you are running EnterpriseOne applications in the latest tools set, there is various memory diagnostic tools. Even callobject kernel log (namely, jdedebug.log) is able to trace the function which cause memory leak.
It is important to fix this memory leak issue if you find this type of message in JDE.log because we can't guarantee whether subsequent issue you hit is valid or not

Windows Heap Manager

A heap memory manager is responsible for the management of heap memory. The heap memory manager performs the following fundamental memory operations:

Allocation - performed by malloc and calloc
Deallocation - performed by free
Reallocation - performed by realloc

A heap is a form of memory manager that an application can use to allocate and free memory dynamically. Whenever the need arises to create data structures whose size cannot be determined statically when a program is being built, the program requires some form of dynamic memory management. The heap grows from lower memory addresses to the upper memory addresses, while the stack does the reverse. It is not possible for them to grow into each other, however. Memory is normally allocated in powers of two and when the amount is not known ahead of time and the size of memory does not fit on the stack (automatic memory), the heap is used. Heaps are Windows objects and Windows maintains pools of memory in heaps. A process can contain several heaps, and you allocate memory from one of these heaps. Memory can be requested from the C runtime, the virtual memory manager, or some other forms of private memory managers. The operational requirements determine if programmers use only the process heap and the C library. The initial heap size, which can be zero and is always rounded up to a multiple of the page size, determines how much physical storage (in a paging file) is committed to the heap. That is, the required space allocated from the heap. As a program exceeds the initial size, additional pages are committed automatically up to a maximum size. Having said that, we must know to break these concepts down to the thread execution, as it is normally writing to a block of memory that it does not own that corrupts the state of the memory block. Suffice it to say that pointers are used to reference memory that contains the memory locations to objects. Rather than moving data around, the operating system uses pointers.

When a process is loaded, the heap manager automatically creates a new heap called the default process heap. As the working set grows, its size for any process is constantly adjusted by Windows. These adjustments may occur because of changes in system load and process priority. Every process has a handle table. Whenever a thread opens a resource, a handle is created in the processs handle table that has a value that the thread can reference in order to continually access that resource. An operating resource can be a file object, a registry key, a TCP/UDP port, a device, etc. Whenever a process is hung, the key is to examine the thread call stack. Read this stack from the bottom to the top. Whenever an executing thread writes to a block of memory it does not own, it will likely corrupt the state of that block of memory. An application can crash outright, or partly run and behave strangely.

<Internal_Only>information below is added as suggested by Developer</Internal_Only>

Other Resources for Troubleshooting memory issue

SafetyCheck

BusBuild and Safety Checker to check compile time BSFN thread safety issue
: Prior to service pack 8.95B, the jdeCallObject kernel was a single threaded entity. That meant, among other things, that one and only one application business function (BSFN) could be executing at any single moment in time. Consequently, EnterpriseOne business functions were developed with this contraint/limitation in mind.

With the delivery of service pack 8.95B, both existing functions and future design methodologies have to change. The jdeCallObject kernel is now multi-threaded - meaning that multiple application BSFNs can execute simultaneously. In order to be considered "threadsafe", BSFNs can no longer use global variables and static variables. In addition, BSFNs can not call functions that are non-threadsafe.

SafetyCheck is a static source code analysis tool that scans C source code and header files for non-threadsafe behaviors. Given a source or header file, SafetyCheck will find instances of global variables, static variables, extern declarations and non-threadsafe ANSI C APIs. When an application developer runs SafetyCheck, he/she will see all instances of non-threadsafe behaviors, as well as line numbers and code snippets of source code.

Pattern Match

<Internal_Only>Static Code analysis and pattern matching tool for checking if BSFN code has called the correct pair of tools API (like JDB_OpenTable is followed by JDB_CloseTable)
This program originated as part of the E1 Applications Security Vulnerability Whitebox Testing project to determine places where potential memory leaks might be exploited either for a heap overrun or as part of a denial of service attack.

Pattern Match offers five distinct types of scans:

A counterpart scan that looks for the absence of particular API calls in context;
A loop invariant scan that examines activity inside a do/while loop;
A scan that determines if a NULL “where clause" is used in table fetches;
A scan that looks for problems with ParseNumericString and ParseNumericStringEx APIs, and
A scan that calculates the cyclomatic complexity of each function within a *.c file.

Additionally, all types of scans can read input files via three distinct methods:

As a single explicitly defined filename,
As input from a “project" file,
By reading all files contained within a single explicitly defined directory

Finally, all scans can create output in detail format or in comma-delimited format for importing into Excel. </Internal_Only>

Example of pattern match can be found at <Document 1050172.1> E1: BSFN: Why is /* FP */ Written in Business Function C Source file (Pattern Match)?

/analyze Compiler Option

You need a license to install and use /analyze and all right reserved by Microsoft. Visual Studio 2005 "/analyze" option in BusBuild for fixing compile time code warning for potential memory overwrite.
The "/analyze" option of the Visual Studio 2005 compiler will perform a static analysis of the code being compiled.
Some of the common warnings seen in E1 are

dereferencing null pointers
using uninitialized memory
index is outside of index range

Using with releases prior to 9.0
The "/analyze" option can be used with prior releases by:

Setting the VisualStudioVersion=8 in your jde.ini.
Setting the flags from the Setup Instructions in your jde.ini.

After you have corrected any warnings from the compiler:

Set the VisualStudioVersion flag to a value appropriate for your release.
Delete the .pdb file for your .dll. It can be found in the b9\\bin32 directory. For example: c:\b9\DV8111\bin32\cdist.pdb.
Rebuild your function. If you don't do this, your system may become unstable.

Code Analysis for C/C++ Warnings can be found MSDN website.

Setup “GlobalFlags” or "G-Flag" and “RTC1”

Setup “GlobalFlags” and “RTC1” to debug heap and stack memory overwrite during runtime on windows platform through Microsoft Visual Studio.
G-Flag (Global Flag)

This debugging tools belong to Microsoft and useful for Windows Platform Server only. GFlags (gflags.exe), the Global Flags Editor, enables and disables advanced internal system diagnostic and troubleshooting features. You can run GFlags from a Command Prompt window or use its graphical user interface dialog box.
Debugger: Specify that a particular program always runs in a debugger. This setting is stored in the registry. It is effective immediately and remains effective until you change it. (This feature is available only in the Global Flags dialog box.)

\TRC1

Compiler Options
Debug Mode: when building in debug mode and using compiler Option "\RTC1", the compiler will put a runtime check flag into the compiled DLL. When the logic causes an exception, the exception will automatically will be caught and the JIT Debugger will be activated.
Note: \RTC1 options will detect memory override. This option is recommended in Debug mode build to help weed out the memory over write issues.

Using the memory diagnostics at runtime

Using the memory diagnostics at runtime to find out potential jdeCache, transaction, table and data pointer and database connection leak as well as using JDB Cache for a transaction table with lots of records.

Oracle Technical Network (OTN)

JD Edwards EnterpriseOne

JD Edwards EnterpriseOne Tools Release 9.1

EnterpriseOne Tools Server and Workstation Administration Guide

Chapter 13 Troubleshooting the Enterprise Server

Troubleshooting additional

In case you need to create your own business function, make it sure that below check points read and executed.

Best Practices C (C Programming Best Practice)

This part of information is to explain possible cause of memory corruption in C programming.

Is the function that allocates memory also the function that frees that memory?

This is just a good coding practice. The function that allocates memory is the function responsible for freeing memory. Any other usage leads to problems where memory is either freed prematurely or (worse) not freed at all.

Are variables declared and initialized at the same time?

Variables contain garbage data when declared on the stack. You cannot make any assumption about what that data looks like until after you have explicitly initialized the variable. For safety reasons, it is always better to initialize the variable at the time it is declared.

Are data structures memset to appropriate size?

Be careful that you’re allocating the correct amount of memory. Always allocate based on the size of a data element and never based on the size of a pointer!

Are arrays declared to be the appropriate size?

Arrays are hard coded in size and do not dynamically expand in size. Consequently, users can overwrite other data when they attempt to store more items in an array than the array is designed to handle. Make sure that your code does not allow for writing outside the allowable bounds of the array.

Are select structures declared to be the appropriate size?

Size your select structures based on the maximum possible of elements that can be used for data selection.

Are pointers stored and retrieved using jdeStoreDataPointer and jdeRetrieveDataPointer? Are they being cleared from the hash table using jdeRemoveDataPointer?

You cannot pass a pointer from a C business function to an ER-based application (natively). The reason for this is that a pointer may have a different size depending on the platform being used and ER does not support variable length parameters. Use jdeStoreDataPointer, jdeRetrieveDataPointer and jdeRemoveDataPointer to handle the passing of pointers/handles to and from ER-based applications. See the JDE API documentation for more details on how to actually implement this functionality.

Are pointers tested for NULL before dereferencing them?

Never dereference a pointer without first testing to see if the pointer is NULL. If you attempt to dereference a null pointer, your program will bomb most ungraciously.

Are handles tested for NULL before using them?

In general, it is a good idea to verify that a handle is not NULL before attempting to use it. For example, before attempting to fetch a record from cache, it might make sense to check that the cache handle is valid before calling the fetch API. This gives your function the opportunity to gracefully handle the fact that the cache handle is invalid. It can be easy to get carried away with checking for NULL pointers and handles every time one is used, especially if either is used extensively. This over-abundance of conditionals makes the code difficult to read and more complex than necessary. In general, a reasonable attempt to check for NULL handles should be made (especially in cases where a NULL condition is likely), but don’t sacrifice clarity.

Are you testing to see if jdeMalloc succeeded in allocating memory?

This is a subtle issue, but important. jdeMalloc is called to allocate new memory from the heap. This function works just fine most of the time. However, there is the possibility that all available memory has been exhausted and jdeMalloc will not, in fact, allocate anything. Don’t simply start using a pointer right after calling jdeMalloc without first testing to see if jdeMalloc actually worked.

Memory Corruption - Common Mistakes

Mismatches in size

Double check the variable/parameters. Mismatches in size can cause the call object kernel to crash.
The cache name must not exceed 50 chars as that’s the maximum allowed size. If the length exceeds 50 chars jdeCacheInit() / jdeCacheInitEx() will return with an error.

Column Arrays

When fetching a subset of columns from a table, if the column array is not properly created it can crash the program. Don't believe it is okay just because it works on your local PC. The AS/400 will have trouble with it.
Column Array Definition / Implementation – we typically use column arrays to open up a table for a subset of columns to improve performance. The key parameters involved are:

Array Definition – defines the desired columns Number of Parameters – defines the # of columns identified in the array Data Structure – populated during the fetch operation
The key rules to follow when using a column array are:

The array and corresponding data structure must match.
The data items used must be listed in the same sequence as they fall in the table.
The # of parameters must accurately reflect the number of data items in the array and data structure.
Key Fields should be included in Column array

In the event any of these rules are violated, the table I/O may function properly locally or on a Windows server environment; however it will fail on the Linux and AS/400.
Common mistakes are:

Modifying the array parameters without updating the corresponding data structure
Listing the parameters of the array in a difference order than the data structure
Appending new columns to the end of the array and data structure without considering where the fall in the table
Adding / Removing parameters from the array & data structure without updating the # of parameters value used by the open table

Referencing null pointers

Forgetting to verify that your pointer is not null before using it.
Using the /analyze Compiler Option can catch these at compile time.

Using uninitialized memory

Forgetting to initialize data structures before use.
Using the /analyze Compiler Option can catch these at compile time for uninitialized structures. It does not seem to catch uninitialized arrays or variables.

Example issue: A business function was doing a JDB_SetSelection on a joined business view. The function hadn't been changed for a while and then it just stopped working on the server. The select structure passed to the JDB_SetSelection was not initialized and contained garbage. It caused the error message "JDB9900269 - Failed to find requested Table -- F34A110" in the jdeDebug.log, but only on a server. It worked perfectly locally.

String Copying

String copies where the destination is too small for the source.
Using the /analyze Compiler Option can catch these at compile time.

Error Message Mapping

jdeCallObject has a parameter for the number of mappings for errors. If you pass a non-zero value but your callmap is null, your system may issue a memory violation.
There is another example of error messages causing memory violations. There was an existing callmap and one of the data structure fields was removed from the function data structure and the callmap was not corrected.

Freeing already freed memory

Freeing the same memory more than once can result in web exceptions. The AS/400 is very unforgiving of freeing the same memory twice.

Mismatches of function parameters

Putting the input parameters in the wrong order can cause web exceptions. For instance a function has parameters A, B, C, D, and E. Passed to the function from the calling function are parameters A, B, D, C, and E. Assuming variables C and D are not the same length and type, this could result in web exceptions.

Spaces in the jdeCallObject function name

This issue was uncovered when researching web exceptions in the 8.10 environment. If there is a space between the function name and the ending quote, a web exception can occur. For example ("GetUDCDefinitions "). This is a very subtle issue that is hard to trace.

Summary:

Memory error may have multiple patterns
It is very important to apply ESUs which are categorized Memory and Performance to have valid result in running EnterpriseOne applications
To minimze possible memory corruption, all the baseline ESUs are to be in place. In general, when there is change in data structure, which is medium for memory communication and which includes business function data structure, processing option template, table data structure, business view data structure, form data structure, report interconnection data structure, media object structure change of size in data dictionary items.
When a certain ESU has dependency with ESU, install all the dependent ESUs to minize any memory error
For debugging purpose,
Check JDE_*.log
Capture JDEDEBUG_*.log
Make use of Memory Diagnostics Tools if applicable
Heap and Stack are handled by Operating system, so logs from EnterpriseOne may not contain full information
If applicable, run offending application in WebDev client with Non-J* environment where DEFAULT BSFN is mapped to LOCAL through OCM. This enables you to determien whether issue is platform (and it's release) specific or not

Known Issues and Bugs:

<Document 1237103.1> E1: 12: R12855 Ends In Memory Violation Error on Windows Server 2008 x64
<Document 632629.1> E1: 34: R3482 & R3483 Common Error Messages and Memory Violations
<Document 639397.1> E1: 30: R30601 EventRule: Business Function BillOfMaterialIntegrityCheckT1
<Document 1064662.1> E1: 43: Possible Causes of Memory Violation in P4314 - Voucher Match Application
<Document 1322258.1> E1: 47: R47011 Reads Memory Violation
<Document 1272974.1> E1: UBE: Case Study on "MEMORY ALLOCATION FAILURE" Errors in EnterpriseOne Batch Applications
<Document 1060818.1> E1: UBE: R98306X Ends in Error and Output is Showing Incomplete Data
<Document 1179493.1> E1: BSFN: Call Object Kernel Errors Clarification - COB0000*

Note:

Oracle's JD Edwards do not warranty any memory corruption issue caused by custom code
This note is to be updated without given any prior notice to indicate more case studies on this topic
This note is based on EnterpriseOne application release 9.1 and it's tools release 9.1
This note is created based on thin clien configuration, which performs business function logic in your logic server

See the second document in this series <Document 1554611.1> E1: BSFN: Case Study on Memory Violation, Memory Corruption, Memory Overwrite, Access Violation and Zombie Kernel Caused by Standard Business Functions Part 2.

Memory Violation Part I

Purpose of Document

Table of Contents

Memory Error

Memory Leak

Memory Corruption

Memory Corruption and Example:

Components for Source File (.C File)

Memory Diagnostic Tools in EnterpriseOne:

Case Study in Interactive Application:

Memory violation in Batch Application:

Open Table Not Closed:

Function Cleanup:

JDB_FreeEnv()

JDE INI Parameters which ease you to debug:

Other issues - Max Cursor and Max Data Pointer:

Windows Heap Manager

Other Resources for Troubleshooting memory issue

SafetyCheck

Pattern Match

/analyze Compiler Option

Setup “GlobalFlags” or "G-Flag" and “RTC1”

Using the memory diagnostics at runtime

Troubleshooting additional

Best Practices C (C Programming Best Practice)

Memory Corruption - Common Mistakes

Summary:

Known Issues and Bugs: