Institute of Natural and Mathematical Sciences

Permanent URI for this communityhttps://mro.massey.ac.nz/handle/10179/4224

Browse

Filters

2002 - 20052

Settings

Subject: search.filters.subject.Data mining

Search Results

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Sparse cross-products of metadata in scientific simulation management
    (Massey University, 2005) James, H.A.; Hawick, K.A.
    Managing scientific data is by no means a trivial task even in a single site environment with a small number of researchers involved. We discuss some issues concerned with posing well-specified experiments in terms of parameters or instrument settings and the metadata framework that arises from doing so. We are particularly interested in parallel computer simulation experiments, where very large quantities of warehouse-able data are involved. We consider SQL databases and other framework technologies for manipulating experimental data. Our framework manages the the outputs from parallel runs that arise from large cross-products of parameter combinations. Considerable useful experiment planning and analysis can be done with the sparse metadata without fully expanding the parameter cross-products. Extra value can be obtained from simulation output that can subsequently be data-mined. We have particular interests in running large scale Monte-Carlo physics model simulations. Finding ourselves overwhelmed by the problems of managing data and compute ¿resources, we have built a prototype tool using Java and MySQL that addresses these issues. We use this example to discuss type-space management and other fundamental ideas for implementing a laboratory information management system.
  • Thumbnail Image
    Item
    Data mining in the survey setting: why do children go off the rails?
    (Massey University, 2002) Scheffer, Judi
    Data Mining is relatively new in the field of statistics, although widely used elsewhere. Is it a good idea to discard the model-based methods in favour of Data Driven methods? Data driven methods produce a high degree of accuracy, but very little interpretability. Model based methods are interpretable, but lack accuracy. Data mining techniques are commonly used where the data collection has been automated. I will show these methods are also useful in the large survey setting.