Infrastructure Software

I'm running OpenSolaris 2008.11 snv_99 on an x86 box with an AMD 64 processor with 2 GB main memory.
Getrlimit returns RLIM_INFINITY when the value of maximum stack size (RLIMIT_STACK) is
131356 KB or greater. I'm using the Sun Studio 12 C compiler. Program demonstrating the problem is
included below. When I run the same program on a Solaris 10 SPARC machine, it works as expected
for max stack size values > 131356 KB, all the way up to 2147483644 bytes for a max stack size.

When the program pauses, if I run "prctl -n process.max-stack-size PID" on the appropriate process ID,
it shows that the setrlimit call apparently worked, but getrlimit returns the wrong value. I also observed
that most shells (sh, csh, tcsh, bash) exhibit the same problem but a few do not (ksh, ksh93). I tried
64-bit versions of these system calls (setrlimit64 and getrlimit64) but it didn't make any difference.

The other strange thing is that a simple test program that just recurses a specified number of times
runs out of stack space and gets a seg fault when the stack size reaches about 131356 KB. Again, this
happens only on x86, not on SPARC.

I searched the bug databases and couldn't find anything similar. Can anyone shed any light on this?

Compilation command: cc -o test6 -D_LARGEFILE64_SOURCE=1 test6.c
Execution results (134508544 = 131356 KB):

ermine% ./test6 134508544
Soft limit = 2147483647 (Infinity)
Hard limit = 2147483647 (Infinity)
Paused:
Soft limit = -3 (Infinity)
Hard limit = -3 (Infinity)
Soft limit = 2147483647 (Infinity)
Hard limit = 2147483647 (Infinity)
Paused:


Source code for test program:

#include <stdio.h>
#include <stdlib.h>
#include <sys/resource.h>

void show_limits(struct rlimit limits);
void show_limits64(struct rlimit64 limits);

char Infinity[] = "(Infinity)";

int main (int argc, char ** argv) {
struct rlimit limits;
struct rlimit64 limits64;
long val;
int rc;
char s[80];

val = atol(argv[1]);

limits.rlim_cur = val;
limits.rlim_max = RLIM_INFINITY;
rc = setrlimit(RLIMIT_STACK, &limits);
if (rc == -1) {
perror("setrlimit failed");
exit(1);
}
rc = getrlimit(RLIMIT_STACK, &limits);
if (rc == -1) {
perror("getrlimit failed");
exit(1);
}
show_limits(limits);
printf("Paused: ");
gets(s);

limits64.rlim_cur = val;
limits64.rlim_max = RLIM64_INFINITY;
rc = setrlimit64(RLIMIT_STACK, &limits64);
if (rc == -1) {
perror("setrlimit64 failed");
exit(1);
}
rc = getrlimit64 (RLIMIT_STACK, &limits64);
if (rc == -1) {
perror("getrlimit64 failed");
exit(1);
}
show_limits64(limits64);
show_limits(limits);
printf("Paused: ");
gets(s);

return 0;
}

void show_limits(struct rlimit limits) {
printf("Soft limit = %ld %s\n", (long) limits.rlim_cur,
limits.rlim_cur == RLIM_INFINITY ? Infinity: "");
printf("Hard limit = %ld %s\n", (long) limits.rlim_max,
limits.rlim_max == RLIM_INFINITY ? Infinity: "");
}

void show_limits64(struct rlimit64 limits) {
printf("Soft limit = %ld %s\n", (long) limits.rlim_cur,
limits.rlim_cur == RLIM64_INFINITY ? Infinity: "");
printf("Hard limit = %ld %s\n", (long) limits.rlim_max,
limits.rlim_max == RLIM64_INFINITY ? Infinity: "");
}