Problems with extremely large scenario files

After running a scenario in NAADSM/PC for several consecutive days, a user recently encountered the following error message:

EXCEPTION: Invalid argument

After some troubleshooting, it turns out that the problem had to do with the extremely large amount of data that was being generated and written to the scenario file. NAADSM uses a Microsoft Access database as the basis of its scenario file, and Access databases are limited to a total file size of 2 GB. When that limit is exceeded, NAADSM will generate an error message similar to the one shown above.

Users can take some steps to prevent this situation from occurring, by (1) not saving unnecessary data or (2) running several separate copies of the same scenario with fewer iterations. These two options are described in more detail below.

Limiting the amount of stored data

Under Scenario parameters -> Output options, there are several settings that allow users to select how much "extraneous" data is stored by NAADSM. One setting allows users to select the number of iterations for which to save daily data. As NAADSM runs, it generates a great deal of information for every simulated day. At the end of each iteration, this information is stored in a more useful summary form, and raw daily data is generally discarded. Unless there is a specific need to save all daily data for every iteration, users should select the option "Save all daily output for a specified number of iterations", and select the minimum number of iterations (currently 3) for which to save this information. If there is a specific research need that requires access to information generated for each simulation day, users should contact a member of the NAADSM Development Team (particularly Aaron Reeves) to discuss alternative approaches.

Also on the Output options screen are settings that will save information for every single event (infection, detection, destruction, etc.) or for every single exposure that occurs during an iteration. Except during scenario troubleshooting, it should virtually never be necessary to store this information. Users should check that these two options are disabled to dramatically reduce the size of files generated by NAADSM.

Running several copies of a scenario

As the number of iterations of a scenario grows, so will the size of the resulting file. A simulation file containing data for 500 iterations will be roughly five times the size of a file containing data for 100 iterations. To reduce file size, it often makes sense to run several copies of the same scenario file with fewer iterations. Because each iteration operates entirely independently of the others, results of 100 iterations each stored in five separate files will be functionally equivalent to the results of 500 iterations stored in a single file. Data from separate files can then be combined for analysis, for example, by using a tool like NaadsmDataToExcel. Note: if you choose to divide files like this, always use the option to automatically generate a seed value for the random number generator. This option can be found under Scenario parameters -> Start setup.

A third option: Using NAADSM/SC

If you anticipate that you will have very large needs for data and storage, it may make more sense to use the supercomputer version of NAADSM, rather than the version for individual PCs. These versions are identical in how they operate and in what information they produce, but the supercomputer version is more practical for very large simulations or datasets. Members of the NAADSM Development Team (particularly Neil Harvey or Shaun Case) may be able to assist you if you think you might require NAADSM/SC.