Error with more than 2^31-1 observations

[mcaceresb]

A bug in Stata causes gtools to exit with error if the user has more than 2^31-1 observations in memory. See this bug report.

I contacted StataCorp about it and they replied:

The SPI can work with datasets containing up to 2^31-1 observations. Our development group is looking into modifying future versions of the SPI to allow more observations.

[mcaceresb]

Re-open because technically it hasn't been fixed. It just throws the correct error now.

[davidmuhlestein]

Any update on this bug - is it correctable now?

[mcaceresb]

@miserton Not afaik. I would contact StataCorp to ask about updates to this point, since I cannot fix it until they update the SPI. Sorry!

[davidmuhlestein]

I reached out to StataCorp:

The stplugin.h has been frozen so that we do not make any changes that could break commands that are still using this code. For 64-bit integer support we recommend working with Java or Python code instead of the older 32-bit C plugin limits.

Unless gtools could call the C code through Java or Python, it doesn't look like the 2.1B limitation will be overcome.

[mcaceresb]

@miserton That's a bit of a strange answer since changing stplugin.h in particular would only involve adding a long long type (and in my system long is 64-bit anyway). I always thought the issue were the internals, and that chancing the internal function to take 64-bit input was difficult for some reason. But I don't really know.

In any case, you are right this is unlikely to be resolved any time soon, if ever.

[wbuchanan]

@miserton can you share who provided that answer to you? @mcaceresb do you have any interest in porting things over to Java or any idea how much different the performance would be in Java compared to C? Also, do you know if C allows overloading function signatures (thinking that might make it even easier since multiple dispatch would prevent breaking any existing code while allowing the same function names to be used with updated types).

[mcaceresb]

@wbuchanan Porting to Java would take quite a bit of effort (partly since I'm not particularly familiar wit Java; not sure if there would be a simple thin wrapper to execute C code in Java but I expect it would still take some work).

The problem is that I cannot modify observations past 2^31-1. The only solution I can think of given the current limitations would be to create multiple separate datasets and append them at the end from Stata.

[davidmuhlestein]

@miserton can you share who provided that answer to you? @mcaceresb do you have any interest in porting things over to Java or any idea how much different the performance would be in Java compared to C? Also, do you know if C allows overloading function signatures (thinking that might make it even easier since multiple dispatch would prevent breaking any existing code while allowing the same function names to be used with updated types).

I got the answer from Stata technical support (tech-support@stata.com).

[wbuchanan]

@mcaceresb that is what I tried the most recent time and there were still too many observations. There is something called the Java Native Interface, but I've not used that at all. However, even though it's been a while I'm still pretty familiar with the Java API for Stata. The only other thing that might be feasible would be trying to figure out how to implement things in a Cythonic way and using the Python API for the entry point, but I'm not sure how much performance degradation that would cause.

[mcaceresb]

@wbuchanan

that is what I tried the most recent time and there were still too many observations.

Wait, if k is the maximum number of variables that match a given stub, you can reshape long by looping ceil(_N * k / (2^31-1)) chunks of roughly equal size, no?